From 9b2d7d0b2be83f60851f38d06136aef1df8da5ff Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:10:49 +0000 Subject: [PATCH 01/44] fix(test): a later passing @expect no longer clears an earlier failure (ako/mxcli#1753) Symptom, from the issue: "On the after-startup test runner, a test with more than one `@expect` is reported **PASS** when an earlier assertion fails and a later one passes. `mxcli test` exits 0." Affected: every Docker run and `--local --legacy-runner`; the endpoint runner is not. Cause: writeExpectAssertion logs one verdict line per assertion, MXTEST:PASS: or MXTEST:FAIL::..., and ParseLogResults let every line overwrite the test's status. The last line won, and the PASS case also cleared the failure message. The generated flow returned $AllPassed = false, but nothing reads it. Fix: the parser folds a test's verdict lines with a sticky failure (applyVerdictLine). Once a test has a FAIL or an ERROR, no later PASS or SKIP line clears it. The first failure's message is reported, with "(and N more failure(s) in this test)" when more followed. This matches the endpoint runner, which evaluates an assertion only while $Verdict is PASS and whose skipped-assertions guard never replaces a failure. The skipped-assertions FAIL (#1448), logged after the PASS lines of the assertions that ran, still wins over them: a sticky failure, not "first line wins". A RUN line resets the fold, so a failure cannot outlive its run. The fix is in the parser, so it also covers logs written by runners generated before it. The generator is unchanged: logging one verdict per test would change a flow whose verdicts were measured on 11.13.0, and it would cover nothing more, because parser and generator ship together. Measurement (unit level, the symptom's layer: the parser reads a log). sticky_failure_test.go, run before the fix existed and again with results.go reverted: --- FAIL: TestLaterPassDoesNotClearEarlierFailure FAIL then PASS: status = PASS, want FAIL FAIL then PASS: message = "", want "Expected $r = 1, actual: 2" FAIL then PASS: the suite passed, so `mxcli test` exits 0 --- FAIL: TestFirstFailureMessageIsReported message = "Expected $r = 4, actual: 2", want "Expected $r = 1, actual: 2 (and 2 more failures in this test)" --- FAIL: TestLaterPassDoesNotClearEarlierError ERROR then PASS:test_1: got PASS "", want ERROR "Setup failed: Shop.Seed" TestSkippedAssertionsFailureStillWins and the PASS-then-FAIL and PASS-then-PASS cases pass both before and after; they are the controls. With the RUN reset stubbed out, TestRunLineStartsVerdictAfresh fails. mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl is the runtime reproduction. It was not run against an app for this change. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...later-expect-pass-clears-earlier-fail.json | 1 + CHANGELOG.md | 1 + cmd/mxcli/testrunner/results.go | 125 ++++++++-------- cmd/mxcli/testrunner/sticky_failure_test.go | 135 ++++++++++++++++++ docs-site/src/tools/test-annotations.md | 4 + ...-runner-1753-earlier-expect-fails.test.mdl | 55 +++++++ 6 files changed, 258 insertions(+), 63 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-later-expect-pass-clears-earlier-fail.json create mode 100644 cmd/mxcli/testrunner/sticky_failure_test.go create mode 100644 mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-later-expect-pass-clears-earlier-fail.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-later-expect-pass-clears-earlier-fail.json new file mode 100644 index 000000000..42148292a --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-later-expect-pass-clears-earlier-fail.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "On the after-startup runner (every Docker run, and `--local --legacy-runner`) a test with more than one `@expect` is reported PASS when an earlier assertion fails and a later one passes, and `mxcli test` exits 0. Measured at unit level: log FAIL:test_1:Expected $r = 1, actual: 2 then PASS:test_1 gave status=PASS message=\"\" AllPassed=true. PASS then FAIL reported correctly; the endpoint runner (`--local`, `--attach`, `--watch`) was never affected", "cause": "`writeExpectAssertion` logs a verdict line per assertion (MXTEST:PASS or MXTEST:FAIL), and `ParseLogResults` treated every line as the test's verdict: each case overwrote Status, and the PASS case also cleared Message, so the last line won. The generated flow did compute `$AllPassed = false` and return it — nothing reads that value", "file": "`cmd/mxcli/testrunner/results.go` (`ParseLogResults`, new `applyVerdictLine`)", "insight": "A log protocol that emits several partial verdicts per test needs a monotonic fold, not last-write-wins: FAIL/ERROR is sticky, the first failure's message is reported (the endpoint runner's semantics — its assertions run only while $Verdict is PASS and its skipped-assertions guard never replaces a failure), and further failures are counted onto the message. The plausible wrong turn is 'first verdict line wins': the skipped-assertions guard (#1448) logs its FAIL AFTER the PASS lines of the assertions that did run, so first-wins turns that false green back on — sticky failure, not first line, is the rule. Fixed in the parser rather than the generator: it also corrects logs from runners generated before the fix, and a one-verdict-per-test generator would change a flow shape whose verdicts were measured on 11.13.0 for no additional coverage, since parser and generator ship in one binary. A RUN line resets the fold so a failure cannot outlive the run it was logged in. Second time this parser took a verdict from the wrong line (after #1448's RUN-without-verdict read as the zero status, PASS): when touching `ParseLogResults`, enumerate the line ORDERS the generator can produce per test, not just the line kinds", "refs": ["ako/mxcli#1753", "ako/mxcli#1448"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 0808e4c63..03eb4926e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. +- **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. - **A button's workflow action survives a rewrite** (ako/mxcli#832) — `describe` printed no action for Show user task page, Show workflow admin page or Call workflow, so re-executing a page turned them into no action (on ako/TestApp's TaskInbox and TaskDashboard, 2 of 6 user-task buttons each). They are now `open user task [with (AssignOnOpen, OpenWhenAssigned)]`, `open workflow with (DefaultPage: M.Page)` and `call workflow M.WF [with (…)]`. `complete task` takes `with (ClosePage, Commit)` instead of always writing both true. `open workflow` without a page is CE1834 and is refused. diff --git a/cmd/mxcli/testrunner/results.go b/cmd/mxcli/testrunner/results.go index 905b232d9..593574229 100644 --- a/cmd/mxcli/testrunner/results.go +++ b/cmd/mxcli/testrunner/results.go @@ -189,6 +189,8 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo // Track which tests we've seen results for resultMap := make(map[string]*TestResult) runTimes := make(map[string]time.Time) + // failures counts each test's FAIL and ERROR lines; see applyVerdictLine. + failures := make(map[string]int) scanner := bufio.NewScanner(logReader) for scanner.Scan() { @@ -233,6 +235,9 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo // verdict did not finish: an error left the runner flow, which // stops it here — a call whose own handler re-raises does exactly // that (ako/mxcli#1448). Its PASS or FAIL line replaces this. + // A RUN line starts the test's verdict afresh, so a failure that + // sticks (applyVerdictLine) cannot outlive the run it belongs to. + delete(failures, id) resultMap[id] = &TestResult{ ID: id, Name: name, @@ -240,77 +245,25 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo Message: "started and reported no verdict: an error left the test runner here, which stopped it (the runtime log has the error)", } - case "PASS": - if r, ok := resultMap[id]; ok { - r.Status = StatusPass - r.Message = "" - if t, ok := runTimes[id]; ok { - r.Duration = time.Since(t) - } - } else { - resultMap[id] = &TestResult{ - ID: id, - Name: id, - Status: StatusPass, - } - } - - case "FAIL": + case "PASS", "FAIL", "ERROR", "SKIP": + // ERROR is a setup that threw: the test never ran, so it is neither a + // pass nor a failure. Only the monolithic runner reports this way — + // the endpoint carries the same outcome back as a SETUP verdict. msg := "" if len(parts) >= 4 { msg = parts[3] } - if r, ok := resultMap[id]; ok { - r.Status = StatusFail - r.Message = msg + r, ok := resultMap[id] + if !ok { + r = &TestResult{ID: id, Name: id} + resultMap[id] = r + } + if action == "PASS" || action == "FAIL" { if t, ok := runTimes[id]; ok { r.Duration = time.Since(t) } - } else { - resultMap[id] = &TestResult{ - ID: id, - Name: id, - Status: StatusFail, - Message: msg, - } - } - - case "ERROR": - // A setup that threw: the test never ran, so it is neither a pass nor - // a failure. Only the monolithic runner reports this way — the - // endpoint carries the same outcome back as a SETUP verdict. - msg := "" - if len(parts) >= 4 { - msg = parts[3] - } - if r, ok := resultMap[id]; ok { - r.Status = StatusError - r.Message = msg - } else { - resultMap[id] = &TestResult{ - ID: id, - Name: id, - Status: StatusError, - Message: msg, - } - } - - case "SKIP": - msg := "" - if len(parts) >= 4 { - msg = parts[3] - } - if r, ok := resultMap[id]; ok { - r.Status = StatusSkip - r.Message = msg - } else { - resultMap[id] = &TestResult{ - ID: id, - Name: id, - Status: StatusSkip, - Message: msg, - } } + applyVerdictLine(r, verdictStatus[action], msg, failures) case "END": result.Duration = time.Since(result.Started) @@ -337,6 +290,13 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo } res.Status = r.Status res.Message = r.Message + if more := failures[tc.ID] - 1; more > 0 { + noun := "failures" + if more == 1 { + noun = "failure" + } + res.Message += fmt.Sprintf(" (and %d more %s in this test)", more, noun) + } res.Duration = r.Duration result.Tests = append(result.Tests, res) } else { @@ -351,6 +311,45 @@ func ParseLogResults(logReader io.Reader, suite *TestSuite, requireAssertions bo return result } +// verdictStatus maps a log protocol verdict action to the status it reports. +var verdictStatus = map[string]TestStatus{ + "PASS": StatusPass, + "FAIL": StatusFail, + "ERROR": StatusError, + "SKIP": StatusSkip, +} + +// applyVerdictLine records one verdict line of the after-startup runner's log +// against a test's result. failures counts each test's FAIL and ERROR lines. +// +// A test can log more than one verdict line: writeExpectAssertion logs PASS or +// FAIL for every @expect on its own, and the skipped-assertions guard logs a +// FAIL after them. A failure is therefore sticky — once a test has a FAIL or an +// ERROR, no later line clears it, and the first failure's message is the one +// reported. When every line simply replaced the last, a test whose first +// assertion failed and whose second held reported PASS with the failure message +// erased, and `mxcli test` exited 0 (ako/mxcli#1753). The rule lives here rather +// than in the generated flow so it also holds for a log written by a runner +// generated before it. +// +// This is the endpoint runner's semantics: there an assertion is evaluated only +// while $Verdict is still PASS, and the skipped-assertions guard does not +// replace an existing failure, so its first failure is what it returns. +// +// The ERROR a RUN line leaves as a placeholder is not a failure line; the +// test's own verdict replaces it. +func applyVerdictLine(r *TestResult, status TestStatus, msg string, failures map[string]int) { + failed := status == StatusFail || status == StatusError + if failed { + failures[r.ID]++ + } + if failures[r.ID] > 1 || (!failed && failures[r.ID] > 0) { + return + } + r.Status = status + r.Message = msg +} + // PrintResults writes a human-readable summary to the writer. func PrintResults(w io.Writer, result *SuiteResult, color bool) { fmt.Fprintf(w, "\n") diff --git a/cmd/mxcli/testrunner/sticky_failure_test.go b/cmd/mxcli/testrunner/sticky_failure_test.go new file mode 100644 index 000000000..993c94a1f --- /dev/null +++ b/cmd/mxcli/testrunner/sticky_failure_test.go @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "strings" + "testing" +) + +// twoExpectSuite is one test with two @expect lines, the shape the after-startup +// runner reports as two verdict lines: writeExpectAssertion logs PASS or FAIL +// for each assertion on its own. +func twoExpectSuite(t *testing.T) *TestSuite { + t.Helper() + e1, err := ParseExpect("$r = 1") + if err != nil { + t.Fatalf("ParseExpect: %v", err) + } + e2, err := ParseExpect("$r = 2") + if err != nil { + t.Fatalf("ParseExpect: %v", err) + } + tc := TestCase{ID: "test_1", Name: "two expects", MDL: "$r = 2;", Expects: []Expect{e1, e2}} + return &TestSuite{Name: "S", Tests: []TestCase{tc}} +} + +// verdictLog is a runner log for test_1 with the given verdict lines between +// its RUN line and the suite's END. +func verdictLog(lines ...string) string { + var b strings.Builder + b.WriteString("MXTEST: MXTEST:START:S\n") + b.WriteString("MXTEST: MXTEST:RUN:test_1:two expects\n") + for _, l := range lines { + b.WriteString("MXTEST: MXTEST:" + l + "\n") + } + b.WriteString("MXTEST: MXTEST:END:S\n") + return b.String() +} + +// TestLaterPassDoesNotClearEarlierFailure is ako/mxcli#1753: on the after-startup +// runner a test with more than one @expect reported PASS when an earlier +// assertion failed and a later one held, and `mxcli test` exited 0. Each +// assertion logs its own verdict line, and the parser let every line overwrite +// the test's status — the last one won, and a PASS also cleared the failure +// message. The endpoint runner evaluates an assertion only while the verdict is +// still PASS, so there the first failure sticks; the log parser now does the +// same. +func TestLaterPassDoesNotClearEarlierFailure(t *testing.T) { + const want = "Expected $r = 1, actual: 2" + + got := ParseLogResults(strings.NewReader(verdictLog("FAIL:test_1:"+want, "PASS:test_1")), + twoExpectSuite(t), false) + r := got.Tests[0] + if r.Status != StatusFail { + t.Errorf("FAIL then PASS: status = %v, want FAIL", r.Status) + } + if r.Message != want { + t.Errorf("FAIL then PASS: message = %q, want %q", r.Message, want) + } + if got.AllPassed() { + t.Error("FAIL then PASS: the suite passed, so `mxcli test` exits 0") + } + + // Control: the order the old parser already got right. + got = ParseLogResults(strings.NewReader(verdictLog("PASS:test_1", "FAIL:test_1:"+want)), + twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusFail || r.Message != want { + t.Errorf("control, PASS then FAIL: got %v %q, want FAIL %q", r.Status, r.Message, want) + } + + // Control: every assertion held. + got = ParseLogResults(strings.NewReader(verdictLog("PASS:test_1", "PASS:test_1")), + twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusPass || r.Message != "" { + t.Errorf("control, PASS then PASS: got %v %q, want PASS", r.Status, r.Message) + } +} + +// TestFirstFailureMessageIsReported pins which message a test that failed more +// than once reports: the first, as the endpoint runner's $Verdict does, with a +// count of the rest so the reader knows the log holds more than the summary. +func TestFirstFailureMessageIsReported(t *testing.T) { + got := ParseLogResults(strings.NewReader(verdictLog( + "FAIL:test_1:Expected $r = 1, actual: 2", + "FAIL:test_1:Expected $r = 3, actual: 2", + "PASS:test_1", + "FAIL:test_1:Expected $r = 4, actual: 2", + )), twoExpectSuite(t), false) + r := got.Tests[0] + if r.Status != StatusFail { + t.Fatalf("status = %v, want FAIL", r.Status) + } + const want = "Expected $r = 1, actual: 2 (and 2 more failures in this test)" + if r.Message != want { + t.Errorf("message = %q, want %q", r.Message, want) + } +} + +// TestSkippedAssertionsFailureStillWins pins the guard writeTestBlock emits after +// the assertions: when a call's own handler resumed the flow past some of them, +// the assertions that did run log PASS and the guard logs a FAIL after them +// (ako/mxcli#1448). That FAIL is the first failure, so it is the verdict. +func TestSkippedAssertionsFailureStillWins(t *testing.T) { + got := ParseLogResults(strings.NewReader(verdictLog( + "PASS:test_1", + "FAIL:test_1:"+assertionsSkippedReason, + )), twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusFail || r.Message != assertionsSkippedReason { + t.Errorf("got %v %q, want FAIL with the skipped-assertions reason", r.Status, r.Message) + } +} + +// TestLaterPassDoesNotClearEarlierError: a setup that threw logs ERROR, and no +// verdict line after it may turn the test into a pass or a skip either. +func TestLaterPassDoesNotClearEarlierError(t *testing.T) { + const msg = "Setup failed: Shop.Seed" + for _, later := range []string{"PASS:test_1", "SKIP:test_1:skipped"} { + got := ParseLogResults(strings.NewReader(verdictLog("ERROR:test_1:"+msg, later)), + twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusError || r.Message != msg { + t.Errorf("ERROR then %s: got %v %q, want ERROR %q", later, r.Status, r.Message, msg) + } + } +} + +// TestRunLineStartsVerdictAfresh: a failure sticks for the run it was logged in, +// not for the rest of the log. A RUN line begins a test's verdict, so a log that +// holds the same test twice reports the later run. +func TestRunLineStartsVerdictAfresh(t *testing.T) { + log := verdictLog("FAIL:test_1:Expected $r = 1, actual: 2", "RUN:test_1:two expects", "PASS:test_1", "PASS:test_1") + got := ParseLogResults(strings.NewReader(log), twoExpectSuite(t), false) + if r := got.Tests[0]; r.Status != StatusPass { + t.Errorf("got %v %q, want the second run's PASS", r.Status, r.Message) + } +} diff --git a/docs-site/src/tools/test-annotations.md b/docs-site/src/tools/test-annotations.md index 4cd744b27..65f045448 100644 --- a/docs-site/src/tools/test-annotations.md +++ b/docs-site/src/tools/test-annotations.md @@ -77,6 +77,10 @@ FAIL the board is 81 squares expected length($result) = 81, actual: 27 ``` +A test with several `@expect` lines passes only when every one of them holds, and +it reports the first that does not. The Docker / `--legacy-runner` path evaluates +them all, and adds how many more failed — `(and 1 more failure in this test)`. + `@expect` cannot be combined with `@throws`: a body expected to fail produces no result to assert on, so the combination is refused rather than ignored. diff --git a/mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl b/mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl new file mode 100644 index 000000000..02f7ab684 --- /dev/null +++ b/mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl @@ -0,0 +1,55 @@ +-- ako/mxcli#1753: on the after-startup runner (every Docker run, and --local +-- --legacy-runner) a test with more than one @expect was reported PASS when an +-- earlier assertion failed and a later one held, and `mxcli test` exited 0. +-- Each assertion logs its own PASS or FAIL line, and the log parser let the +-- last line be the test's verdict. A failure now sticks, and the first one is +-- the message reported. +-- +-- Setup: test-runner-1514-setup.mdl (Probe.SUB_Make returns 'a;b'). Then: +-- +-- mxcli test mdl-examples/bug-tests/test-runner-1753-earlier-expect-fails.test.mdl -p app.mpr --local --legacy-runner +-- +-- Expected: test 1 FAIL "Expected $a = 'zzz', actual: a;b" (before the fix: +-- PASS), test 2 FAIL with the same message for $b (the control — the order the +-- parser always got right), test 3 FAIL "Expected $c = 'zzz', actual: a;b (and +-- 1 more failure in this test)", test 4 PASS. The endpoint runner (--local +-- alone) evaluates an assertion only while the test is still passing, so it +-- reported test 1 correctly before the fix too. +-- +-- The log parser's verdicts are pinned by cmd/mxcli/testrunner/ +-- sticky_failure_test.go; this file is the runtime reproduction and was not run +-- against an app for the fix. Each test uses its own variable so the file does +-- not depend on the runner renaming a shared one. +mdl 1; +/** + * @test 1 earlier assertion fails, later one holds + * @expect $a = 'zzz' + * @expect $a = 'a;b' + */ +$a = call microflow Probe.SUB_Make(); +/ + +/** + * @test 2 earlier assertion holds, later one fails + * @expect $b = 'a;b' + * @expect $b = 'zzz' + */ +$b = call microflow Probe.SUB_Make(); +/ + +/** + * @test 3 two assertions fail, one holds + * @expect $c = 'zzz' + * @expect $c = 'a;b' + * @expect $c = 'yyy' + */ +$c = call microflow Probe.SUB_Make(); +/ + +/** + * @test 4 every assertion holds + * @expect $d = 'a;b' + * @expect length($d) = 3 + */ +$d = call microflow Probe.SUB_Make(); +/ From be9dc02389364c7a6969bb3ba361ce1b2d2e9fac Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:15:45 +0000 Subject: [PATCH 02/44] fix(test): rename a lower-case `call`/`create` variable per test on the after-startup runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom, from ako/mxcli#1753's sibling note: "on this runner, `assignPattern` matches only upper-case `CALL|CREATE`. So mdl 1's `$r = call …` is not renamed per test, and two tests that both use `$r` fail injection (MDL063 / CE0111, measured)." Affected: every Docker run and `--local --legacy-runner`. The endpoint runner gives each test its own flow and does not rename. Cause: the legacy runner compiles every test into one microflow, so it gives each test's variables a `_N` suffix. assignPattern decides which variables are renamed, and it matched only `CALL|CREATE` in upper case. MDL keywords are case-insensitive, and mdl 1 and `fmt` write them in lower case, so `$r = call microflow …` stayed `$r` in every test. Fix: assignPattern matches `call|create` in any case, followed by a word boundary so `$r = callback` does not count. No other pattern in cmd/mxcli/testrunner makes the upper-case assumption: the rest lower-case their input or use EqualFold. TestThrowsCatchesEveryCallKind and TestVerdictCatchesEveryCallKind matched the legacy output against the unrenamed lower-case statement, so they passed only because of the defect. Their legacy assertions now expect the renamed statement (legacyStmt). The 1448 bug-test's note that the legacy runner "needs a distinct variable name per test" is updated. Measurement. rename_keyword_case_test.go with the old pattern (the upper-case control subtest passes): --- FAIL: TestLegacyRunnerRenamesLowerCaseAssignments/call_microflow,_mdl_1 test 1: no "$r_1 = call microflow" — the assignment was not renamed an assertion still reads the unrenamed $r MDL063: '$r' is created twice in this microflow — first by call microflow, then by call microflow. ... mxbuild rejects this with CE0111 "Duplicate variable name" (and the same for mdl 0, call java action, create object, create list, mixed case) --- FAIL: TestAssignPatternNeedsAWholeKeyword control: $r = call microflow was not picked up: map[] I also swept every non-.fail `.test.mdl` in mdl-examples/ (16 files) through GenerateTestRunner and executor.ValidateProgram. Before the fix, 10 files drew MDL063, among them the 1448, 1514 and 923 bug-tests and doctype-tests/microflow-spec (86). After it, 2 do. Those 2 come from `retrieve $Brands` and from `retrieve $Request` / `$Request = head(…)`. They are a separate gap: assignPattern has never matched `retrieve`, `declare`, list operations, `send rest request` or `execute database query` in any case. This change does not address it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-lower-case-call-variable-not-renamed.json | 1 + CHANGELOG.md | 1 + cmd/mxcli/testrunner/generator.go | 10 +- .../testrunner/on_error_coverage_test.go | 13 ++- .../testrunner/rename_keyword_case_test.go | 94 +++++++++++++++++++ ...est-runner-1448-own-error-handler.test.mdl | 7 +- 6 files changed, 120 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-lower-case-call-variable-not-renamed.json create mode 100644 cmd/mxcli/testrunner/rename_keyword_case_test.go diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-lower-case-call-variable-not-renamed.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-lower-case-call-variable-not-renamed.json new file mode 100644 index 000000000..cd1b37502 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-legacy-runner-lower-case-call-variable-not-renamed.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "On the after-startup runner (Docker, `--local --legacy-runner`) two tests in one file that both assign the same variable with a lower-case keyword — `$r = call microflow …`, mdl 1's and `fmt`'s spelling — fail injection with MDL063 (CE0111 \"Duplicate variable name\" from mxbuild); the same file runs under the endpoint runner, which gives each test its own flow", "cause": "The legacy runner compiles every test into one microflow and renames each test's variables with a `_N` suffix, but which variables to rename was decided by `assignPattern`, which matched only upper-case `CALL|CREATE` — the spelling of the runner's own early examples. A lower-case assignment was left unrenamed and declared once per test", "file": "`cmd/mxcli/testrunner/generator.go` (`assignPattern`, now `(?mi)…(?:call|create)\\b`)", "insight": "The cheapest detector was a sweep, not a reading: every `.test.mdl` in the repo through `GenerateTestRunner` and `executor.ValidateProgram` — 10 of the 16 in mdl-examples/ drew MDL063 before the fix, 2 after. Those 2 (`retrieve $Brands`, `retrieve $Request` + `$Request = head(…)`) are the remaining, separate gap: `assignPattern` never matched `retrieve $x`, `declare $x`, list operations (`$x = filter(…)`), `send rest request` or `execute database query` in ANY case, so a variable those define in two tests still collides; the durable fix is to take the declared names from the parsed body (`visitor.FlowStatements`, as `attachOnError` already does), not a regex. Two existing tests (`TestThrowsCatchesEveryCallKind`, `TestVerdictCatchesEveryCallKind`) matched the legacy output against the unrenamed lower-case statement and passed only because of the defect — a test that asserts a generator's text verbatim can encode the bug it should catch. The `\\b` matters once the match is case-insensitive (`$r = callback`). Found alongside ako/mxcli#1753, whose bug-test therefore uses a distinct variable per test", "refs": ["ako/mxcli#1753"], "rules": ["MDL063"], "ce": ["CE0111"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 03eb4926e..093bf72ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. +- **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. - **A button's workflow action survives a rewrite** (ako/mxcli#832) — `describe` printed no action for Show user task page, Show workflow admin page or Call workflow, so re-executing a page turned them into no action (on ako/TestApp's TaskInbox and TaskDashboard, 2 of 6 user-task buttons each). They are now `open user task [with (AssignOnOpen, OpenWhenAssigned)]`, `open workflow with (DefaultPage: M.Page)` and `call workflow M.WF [with (…)]`. `complete task` takes `with (ClosePage, Commit)` instead of always writing both true. `open workflow` without a page is CE1834 and is refused. diff --git a/cmd/mxcli/testrunner/generator.go b/cmd/mxcli/testrunner/generator.go index 674b81b3d..0e347a7f9 100644 --- a/cmd/mxcli/testrunner/generator.go +++ b/cmd/mxcli/testrunner/generator.go @@ -273,9 +273,15 @@ func writeExpectAssertion(b *strings.Builder, testID string, exp Expect, evaluat // varPattern matches $VariableName in MDL ($ followed by word characters). var varPattern = regexp.MustCompile(`\$([A-Za-z_][A-Za-z0-9_]*)`) -// assignPattern matches "$var = CALL" or "$var = CREATE" at the start of any +// assignPattern matches "$var = call …" or "$var = create …" at the start of any // statement line inside the test body. -var assignPattern = regexp.MustCompile(`(?m)^\s*\$([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(?:CALL|CREATE)`) +// +// The keywords match in any case. MDL keywords are case-insensitive, and mdl 1 +// and `fmt` write them in lower case; matching only `CALL|CREATE` left +// `$r = call microflow …` unrenamed, so two tests that both assigned `$r` declared +// it twice in the one runner flow and injection failed with MDL063 (CE0111). +// The `\b` keeps a value that merely starts with those letters from matching. +var assignPattern = regexp.MustCompile(`(?mi)^\s*\$([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(?:call|create)\b`) // extractVariableNames finds all user-defined variable names in the MDL body. // Returns the set of variable names (without $ prefix). diff --git a/cmd/mxcli/testrunner/on_error_coverage_test.go b/cmd/mxcli/testrunner/on_error_coverage_test.go index d190814ac..669a89858 100644 --- a/cmd/mxcli/testrunner/on_error_coverage_test.go +++ b/cmd/mxcli/testrunner/on_error_coverage_test.go @@ -36,7 +36,7 @@ func TestThrowsCatchesEveryCallKind(t *testing.T) { t.Run(c.kind, func(t *testing.T) { tc := parseOne(t, "/**\n * @test it raises\n * @throws 'Only http and https csv urls are supported'\n */\n"+c.stmt+"\n/\n") assertHandled(t, "endpoint runner", GenerateTestFlows(&TestSuite{Name: "s", Tests: []TestCase{tc}}), c.stmt) - assertHandled(t, "legacy runner", GenerateTestRunner(&TestSuite{Name: "s", Tests: []TestCase{tc}}), c.stmt) + assertHandled(t, "legacy runner", GenerateTestRunner(&TestSuite{Name: "s", Tests: []TestCase{tc}}), legacyStmt(c.stmt)) }) } } @@ -53,7 +53,7 @@ func TestVerdictCatchesEveryCallKind(t *testing.T) { if !strings.Contains(mdl, verdictFailPrefix+"exception during execution: ") { t.Errorf("no FAIL verdict in the handler:\n%s", mdl) } - assertHandled(t, "legacy runner", GenerateTestRunner(&TestSuite{Name: "s", Tests: []TestCase{tc}}), c.stmt) + assertHandled(t, "legacy runner", GenerateTestRunner(&TestSuite{Name: "s", Tests: []TestCase{tc}}), legacyStmt(c.stmt)) }) } } @@ -78,3 +78,12 @@ func assertHandled(t *testing.T, runner, mdl, stmt string) { t.Errorf("%s: generated MDL does not parse: %v\n%s", runner, errs[0], mdl) } } + +// legacyStmt is stmt as the legacy runner writes it for the suite's first test: +// that runner compiles every test into one flow, so it renames the variable a +// call assigns with the test's suffix. These statements used to be matched +// unrenamed, which held only because a lower-case `call` was not renamed +// (TestLegacyRunnerRenamesLowerCaseAssignments). +func legacyStmt(stmt string) string { + return renameVariables(stmt, extractVariableNames(stmt), "_1") +} diff --git a/cmd/mxcli/testrunner/rename_keyword_case_test.go b/cmd/mxcli/testrunner/rename_keyword_case_test.go new file mode 100644 index 000000000..ef6384724 --- /dev/null +++ b/cmd/mxcli/testrunner/rename_keyword_case_test.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// TestLegacyRunnerRenamesLowerCaseAssignments: the after-startup runner compiles +// every test into one microflow, so a variable two tests both assign has to be +// renamed per test. Which variables are renamed was decided by a pattern that +// matched only upper-case `CALL` and `CREATE` — the spelling the runner's own +// examples used — so `$r = call microflow …`, how mdl 1 and `fmt` spell it, was +// not renamed. Two tests that both assign `$r` then declare it twice, and +// injection fails with MDL063 (CE0111 "Duplicate variable name" from mxbuild). +// Found alongside ako/mxcli#1753. +func TestLegacyRunnerRenamesLowerCaseAssignments(t *testing.T) { + cases := []struct { + name, header, body, renamed string + }{ + {"call microflow, mdl 1", "mdl 1;\n", + "$r = call microflow Probe.SUB_Make();", "$r_%d = call microflow"}, + {"call microflow, mdl 0", "", + "$r = call microflow Probe.SUB_Make();", "$r_%d = call microflow"}, + {"call java action", "mdl 1;\n", + "$r = call java action Probe.JA_Upper(Text = 'a');", "$r_%d = call java action"}, + {"create object", "mdl 1;\n", + "$r = create Probe.Item (Name = 'a');", "$r_%d = create Probe.Item"}, + {"create list", "mdl 1;\n", + "$r = create list of Probe.Item;", "$r_%d = create list of"}, + {"mixed case", "mdl 1;\n", + "$r = Call Microflow Probe.SUB_Make();", "$r_%d = Call Microflow"}, + // Control: the spelling the pattern always matched. + {"upper case (control)", "", + "$r = CALL MICROFLOW Probe.SUB_Make();", "$r_%d = CALL MICROFLOW"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + block := func(n string) string { + return "/**\n * @test " + n + "\n * @expect $r != empty\n */\n" + c.body + "\n/\n" + } + tests, err := parseMDLTests(c.header+block("one")+"\n"+block("two"), "rename.test.mdl") + if err != nil { + t.Fatalf("parse: %v", err) + } + src := GenerateTestRunner(&TestSuite{Name: "s", Tests: tests}) + + for i, want := range []string{ + strings.Replace(c.renamed, "%d", "1", 1), + strings.Replace(c.renamed, "%d", "2", 1), + } { + if !strings.Contains(src, want) { + t.Errorf("test %d: no %q — the assignment was not renamed", i+1, want) + } + } + if strings.Contains(src, "($r != empty)") { + t.Errorf("an assertion still reads the unrenamed $r") + } + + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("generated runner does not parse: %v\n%s", errs[0], src) + } + for _, v := range executor.ValidateProgram(prog, "") { + if v.RuleID == "MDL063" { + t.Errorf("%s: %s", v.RuleID, v.Message) + } + } + if t.Failed() { + t.Logf("generated runner:\n%s", src) + } + }) + } +} + +// TestAssignPatternNeedsAWholeKeyword: matching case-insensitively must not turn +// an assignment whose value merely starts with the letters of a keyword into one. +func TestAssignPatternNeedsAWholeKeyword(t *testing.T) { + for _, body := range []string{ + "$r = callback;", + "$r = createdDate;", + } { + if names := extractVariableNames(body); len(names) != 0 { + t.Errorf("%q: renamed %v, want nothing", body, names) + } + } + if names := extractVariableNames("$r = call microflow M.F();"); !names["r"] { + t.Errorf("control: $r = call microflow was not picked up: %v", names) + } +} diff --git a/mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl b/mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl index 7bf7545c7..1987183ff 100644 --- a/mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl +++ b/mdl-examples/bug-tests/test-runner-1448-own-error-handler.test.mdl @@ -31,8 +31,11 @@ -- -- Under --legacy-runner a test whose error leaves the runner (05, 06, 11) ends -- the whole run; 05 was reported PASS there (a RUN line with no verdict read as --- the zero status) and is now an ERROR. That runner also needs a distinct --- variable name per test. +-- the zero status) and is now an ERROR. That runner also needed a distinct +-- variable name per test: it renamed a variable per test only after an +-- upper-case `CALL`, so this file's `$r` and `$Result` drew 14 MDL063 +-- duplicate-variable errors there. It renames one after a lower-case `call` too +-- since the sibling fix to ako/mxcli#1753. mdl 1; /** * @test 01 plain call that throws From 2f9b6260c78b3ab6b1bdadcefd4d422308dbf62a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:50:32 +0000 Subject: [PATCH 03/44] fix(security): keep a module role's description when create or modify omits it Symptom (ako/mxcli#1218): a bare `create or modify module role M.R;` reported "Modified module role: M.R" and erased the role's stored description. `describe` -> `exec` was unaffected because describe prints the description; grants were untouched. Cause: CreateModuleRoleStmt.Description was a plain string, so "no description clause" and `description ''` were the same value, and the OR MODIFY branch of execCreateModuleRole passed it to AddModuleRole, which overwrites the existing role in place. Fix: Description is a *string, nil when the statement has no clause, as CreateUserRoleStmt's properties have been since ako/mxcli#707. OR MODIFY writes the stored description when it is nil, so the write is elided and the run reports "Unchanged"; `description ''` clears it. Adopting an auto-provisioned role still drops its marker description. Docs (skill, docs-site reference, `mxcli syntax`) state the rule. Measurement: - Unit test (mdl/executor/issue1218_module_role_description_test.go) fails before the fix, and with the fix forced back, with the reported symptom: description written = "", want "Read the published OData services. Nothing else." / Modified module role: Shop.ApiUser. - Real app, Mendix 11.13.0: role created with a description; bare statement with the fixed build -> "Unchanged module role", describe keeps the description, describe -> exec "Unchanged"; with the pre-fix build -> "Modified module role" and the description is empty. `description ''` with the fixed build clears it. mx check: 0 errors in every state, the same as the blank app, fault included. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...e-create-or-modify-erases-description.json | 1 + .../skills/mendix/manage-security/SKILL.md | 12 ++- CHANGELOG.md | 1 + cmd/mxcli/syntax/features_security.go | 2 +- .../reference/security/create-module-role.md | 12 ++- .../1218-module-role-description-kept.mdl | 34 +++++++ mdl/ast/ast_security.go | 15 ++- mdl/executor/cmd_security_write.go | 16 +++- .../issue1218_module_role_description_test.go | 94 +++++++++++++++++++ .../issue707_describe_roundtrip_test.go | 4 +- .../module_role_create_statement_test.go | 2 +- mdl/visitor/visitor_security.go | 5 +- .../visitor_security_or_modify_test.go | 4 +- mdl/visitor/visitor_security_test.go | 25 ++++- 14 files changed, 203 insertions(+), 24 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-28-module-role-bare-create-or-modify-erases-description.json create mode 100644 mdl-examples/bug-tests/1218-module-role-description-kept.mdl create mode 100644 mdl/executor/issue1218_module_role_description_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-module-role-bare-create-or-modify-erases-description.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-module-role-bare-create-or-modify-erases-description.json new file mode 100644 index 000000000..cae865287 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-module-role-bare-create-or-modify-erases-description.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-28","symptom":"A bare `create or modify module role M.R;` (no description clause) reported `Modified module role: M.R` and erased the role's stored Description; `describe` → `exec` was unaffected because describe prints the description, and grants (on the entity, keyed by role name) were untouched.","cause":"CreateModuleRoleStmt.Description was a plain string, so 'no clause' and `description ''` were the same value, and execCreateModuleRole's OR MODIFY branch passed it to Backend.AddModuleRole, which overwrites the existing role in place.","file":"mdl/ast/ast_security.go (Description *string), mdl/visitor/visitor_security.go, mdl/executor/cmd_security_write.go (execCreateModuleRole)","fix":"Description is a *string, nil when unstated, as CreateUserRoleStmt's properties have been since ako/mxcli#707; OR MODIFY writes the stored description when it is nil, so the write is elided and the run reports Unchanged. `description ''` clears it. Adopting an auto-provisioned role still drops its marker description.","insight":"A neighbouring statement already had the right shape: the user-role code even said 'the rest keep their stored values, as a module role's missing description does', which was false — a comment asserting a sibling's behaviour is a claim to test, not evidence. The tell for this class is an optional clause parsed into a value type: grep the AST for optional clauses stored as string/bool without a *T or a ...Set flag, then follow each to an OR MODIFY branch. mx check is 0 errors with the fault (measured, 11.13.0), and describe → exec cannot show it because describe restates the value; only a hand-written minimal statement does.","refs":["ako/mxcli#1218"],"test":"mdl/executor/issue1218_module_role_description_test.go, mdl/visitor/visitor_security_test.go"} diff --git a/.claude/skills/mendix/manage-security/SKILL.md b/.claude/skills/mendix/manage-security/SKILL.md index 9f294a433..e75335f11 100644 --- a/.claude/skills/mendix/manage-security/SKILL.md +++ b/.claude/skills/mendix/manage-security/SKILL.md @@ -13,11 +13,11 @@ description: "Configure Mendix security in MDL — module and user roles, access | 27-35 | When to Use This Skill | | 36-43 | Security Concepts | | 44-120 | The System-module ceiling — decide the role model around this FIRST | -| 121-558 | Syntax Reference | -| 559-573 | Starlark Lint Rule APIs | -| 574-626 | Common Workflow: Setting Up Module Security | -| 627-636 | Common Mistakes | -| 637-646 | Validation | +| 121-560 | Syntax Reference | +| 561-575 | Starlark Lint Rule APIs | +| 576-628 | Common Workflow: Setting Up Module Security | +| 629-638 | Common Mistakes | +| 639-648 | Validation | # Security Management Skill @@ -205,7 +205,9 @@ create module role MyModule.Viewer description 'Read-only access'; -- `or modify` updates an existing role's description instead of failing, so the -- whole security script stays re-runnable rather than needing a run-once file. +-- Without `description` it keeps the stored one; `description ''` clears it. create or modify module role MyModule.ApiUser description 'API consumer'; +create or modify module role MyModule.Admin; -- ensures it exists, keeps its description -- Remove a module role (`if exists` makes it a no-op when the role is gone) drop module role MyModule.Viewer; diff --git a/CHANGELOG.md b/CHANGELOG.md index 093bf72ec..a947aaac4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **`check -p --references` reports CE1406 for a retrieve, create or create list of a Hidden entity of a Protected module** (ako/mxcli#1699, mendixlabs/mxcli#1652) — a flow in an app module that ran `retrieve $l from AddOn.Internal`, `$o = create AddOn.Internal` or `$l = create list of AddOn.Internal` on a Hidden entity of a Protected (add-on) module passed `check` and `exec`, and mxbuild refused it: `[CE1406] "Retrieve object(s) activity 'Retrieve list of Internal from database' is not allowed to use entity 'AddOn.Internal' because app modules cannot reference hidden elements in add-on modules."` (likewise "Create object activity", "Create list activity"). The check judged an entity only as a parameter or return type; it now judges the entity these three activities name, in microflows and nanoflows, and `exec` refuses the script before writing it. Measured on Mendix 11.13.0: each shape refused against a Hidden entity, 0 errors against a Usable one and inside the add-on; an association retrieve that reaches a Hidden entity over a Usable association names only the association and is not refused. - **A pluggable widget's text-template parameter across an association builds, and describes so it re-executes** (mendixlabs/mxcli#1494) — `fieldset fs (legend: '{1}', legendParams: ({1} = M.Parent_Child/Label))` passed `check` and `exec` and failed the build with `[CE1365] "Attribute 'M.Child.Label' is not an attribute of entity 'M.Parent'."`: the parameter was written with its attribute but without the association hop, so mxbuild read the attribute against the widget's own entity. The same writer serves an object-list item's template (an Accordion group's `headerText`) and a Data grid 2 column's dynamic text, and all three failed; on Mendix 11.12.4 they now build with 0 errors. `describe page` printed such a parameter as the bare attribute (`{1} = Label`), which re-executed into CE1613 "The selected attribute 'M.Parent.Label' no longer exists"; it now prints `M.Parent_Child/Label`. The companion report, an expression parameter (`{1} = if 1 = 1 then 'A' else 'B'`) stored as an attribute name (mendixlabs/mxcli#1495), was fixed in v0.25.0 (#823) and is now pinned by a test. - **`alter entity … add event handler` adds a second handler for the same event** (mendixlabs/mxcli#1540) — `add event handler on before commit call M.Second($currentObject)` failed with "event handler already exists for Before Commit" when the entity already had one, though Studio Pro allows several handlers for one event and `create or modify entity` with two `on before commit` clauses stores both (measured on 11.12.4: two Before Commit handlers are 0 errors, a void-returning one is CE7247, so the check does look at them). A handler is now identified by its moment, event and microflow: another microflow adds a second handler, the same microflow again is still a duplicate. The same key had made `add event handler if not exists … call M.New` skip — reporting "already exists" — whenever any handler for the event existed, and `drop event handler on before commit` remove whichever handler was stored first. `drop event handler` now takes `call M.Mf` to say which one; without it, a drop that matches several handlers is refused and names them. +- **A bare `create or modify module role M.R;` keeps the role's description** (ako/mxcli#1218) — the "make sure this role exists" statement reported `Modified module role: M.R` and erased the stored description: a statement without `description` was written as an empty one. An omitted `description` now keeps what is stored, as an omitted user role property already did, and the run reports `Unchanged`; `description ''` clears it. Grants were never affected, and `describe` → `exec` was not, since `describe` prints the description. Measured on Mendix 11.13.0: `mx check` is 0 errors with and without the fault, so only the stored description showed it. ## [0.28.0] - 2026-10-23 diff --git a/cmd/mxcli/syntax/features_security.go b/cmd/mxcli/syntax/features_security.go index 83372b349..698f5a7d4 100644 --- a/cmd/mxcli/syntax/features_security.go +++ b/cmd/mxcli/syntax/features_security.go @@ -22,7 +22,7 @@ func init() { "module role", "create role", "drop role", }, Syntax: "CREATE [OR MODIFY] MODULE ROLE . [DESCRIPTION ''];\nDROP MODULE ROLE .;", - Example: "mdl 1;\nCREATE MODULE ROLE Shop.Admin DESCRIPTION 'Full access';\n-- OR MODIFY makes a security script re-runnable:\nCREATE OR MODIFY MODULE ROLE Shop.User DESCRIPTION 'Read-only access';", + Example: "mdl 1;\nCREATE MODULE ROLE Shop.Admin DESCRIPTION 'Full access';\n-- OR MODIFY makes a security script re-runnable:\nCREATE OR MODIFY MODULE ROLE Shop.User DESCRIPTION 'Read-only access';\n-- Without DESCRIPTION it keeps the stored one; DESCRIPTION '' clears it:\nCREATE OR MODIFY MODULE ROLE Shop.Admin;", SeeAlso: []string{"security.user-role", "security.entity-access"}, }) diff --git a/docs-site/src/reference/security/create-module-role.md b/docs-site/src/reference/security/create-module-role.md index 13ed8b972..6a073d25c 100644 --- a/docs-site/src/reference/security/create-module-role.md +++ b/docs-site/src/reference/security/create-module-role.md @@ -3,7 +3,7 @@ ## Synopsis ```sql -CREATE MODULE ROLE module.RoleName [ DESCRIPTION 'text' ] +CREATE [ OR MODIFY ] MODULE ROLE module.RoleName [ DESCRIPTION 'text' ] ``` ## Description @@ -12,7 +12,7 @@ Creates a new module role within a module. Module roles are the foundation of Me Each module role belongs to exactly one module. The role name must be unique within the module. By convention, common role names include `User`, `Admin`, `Viewer`, and `Manager`. -If the module does not exist, the statement returns an error. +If the module does not exist, the statement returns an error. If the role exists, `CREATE MODULE ROLE` returns an error; `CREATE OR MODIFY` updates it instead, so a security script can be re-run. Without `DESCRIPTION`, `CREATE OR MODIFY` keeps the role's stored description; `DESCRIPTION ''` clears it. ## Parameters @@ -20,7 +20,7 @@ If the module does not exist, the statement returns an error. : Qualified name consisting of the module name and the new role name, separated by a dot. The module must already exist. `DESCRIPTION 'text'` -: Optional human-readable description of the role's purpose. +: Optional human-readable description of the role's purpose. Left out, a new role has none and an existing role keeps its own. ## Examples @@ -45,6 +45,12 @@ CREATE MODULE ROLE HR.Employee DESCRIPTION 'Self-service access'; CREATE MODULE ROLE HR.Viewer DESCRIPTION 'Read-only access'; ``` +Make sure a role exists, keeping whatever description it has: + +```sql +CREATE OR MODIFY MODULE ROLE HR.Manager; +``` + ## See Also [CREATE USER ROLE](create-user-role.md), [GRANT](grant.md), [REVOKE](revoke.md) diff --git a/mdl-examples/bug-tests/1218-module-role-description-kept.mdl b/mdl-examples/bug-tests/1218-module-role-description-kept.mdl new file mode 100644 index 000000000..417838328 --- /dev/null +++ b/mdl-examples/bug-tests/1218-module-role-description-kept.mdl @@ -0,0 +1,34 @@ +mdl 1; +-- ako/mxcli#1218: a bare `create or modify module role M.R;` silently erased +-- the role's description. +-- +-- Reported: +-- $ mxcli exec probe.mdl -p Formula1Backend.mpr +-- Modified module role: Formula1Backend.ApiUser +-- and `show module roles` had the role's Description empty afterwards. The +-- executor handed the statement's empty description to the backend, which +-- overwrites; an omitted DESCRIPTION clause was read as "set it to ''". +-- +-- An omitted clause now keeps what is stored, as an omitted user role property +-- already did (ako/mxcli#707). `description ''` is the spelling that clears it. +-- +-- mxcli exec 1218-module-role-description-kept.mdl -p app.mpr +-- mxcli -p app.mpr -c "show module roles in Zz1218" +-- +-- After the run Zz1218.ApiUser has the description from statement 1, and +-- Zz1218.Cleared has none. Re-running the script reports `Unchanged` for the +-- bare statement and `Modified` only where it states a different description. +-- Measured on Mendix 11.13.0: `mx check` is 0 errors with the fault and +-- without it, so only the stored description shows the difference. +-- The unit test is mdl/executor/issue1218_module_role_description_test.go. + +create or modify module Zz1218; + +create or modify module role Zz1218.ApiUser description 'Read the published OData services. Nothing else.'; + +-- The "make sure this role exists" idiom: keeps the description above. +create or modify module role Zz1218.ApiUser; + +-- Stating an empty description is how a script clears one. +create or modify module role Zz1218.Cleared description 'Temporary'; +create or modify module role Zz1218.Cleared description ''; diff --git a/mdl/ast/ast_security.go b/mdl/ast/ast_security.go index 18c3d38fc..80e6f0328 100644 --- a/mdl/ast/ast_security.go +++ b/mdl/ast/ast_security.go @@ -10,13 +10,26 @@ package ast type CreateModuleRoleStmt struct { CreateGuard // `create … if not exists` (ako/mxcli#731) Name QualifiedName - Description string + // Description is nil when the statement has no DESCRIPTION clause: a new + // role then has none, and `create or modify` keeps the stored one + // (ako/mxcli#1218), as CreateUserRoleStmt's properties do. `description ''` + // is the stated empty description, which clears it. + Description *string // CreateOrModify makes the statement idempotent: an existing role has its // description updated instead of the statement failing, so a security script // can be re-run. CreateOrModify bool } +// DescriptionText is the description the statement states, or "" when it +// states none. +func (s *CreateModuleRoleStmt) DescriptionText() string { + if s.Description == nil { + return "" + } + return *s.Description +} + func (s *CreateModuleRoleStmt) isStatement() {} // DropModuleRoleStmt represents: DROP MODULE ROLE [IF EXISTS] Module.RoleName diff --git a/mdl/executor/cmd_security_write.go b/mdl/executor/cmd_security_write.go index 808f4354a..3b8eccd8a 100644 --- a/mdl/executor/cmd_security_write.go +++ b/mdl/executor/cmd_security_write.go @@ -44,7 +44,9 @@ func execCreateModuleRole(ctx *ExecContext, s *ast.CreateModuleRoleStmt) error { if mr.Description == autoDocumentRoleDescription { oldQualified := s.Name.Module + "." + mr.Name newQualified := s.Name.Module + "." + s.Name.Name - if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.Description); err != nil { + // Adopting the role takes it over: the marker description is not + // the author's documentation, so an unstated description clears it. + if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.DescriptionText()); err != nil { return mdlerrors.NewBackend("create module role", err) } // If the casing actually changed, propagate the rename across every @@ -64,8 +66,14 @@ func execCreateModuleRole(ctx *ExecContext, s *ast.CreateModuleRoleStmt) error { if s.CreateOrModify { // Re-running a security script must not fail on a role that is // already there. AddModuleRole overwrites, so this also adopts a new - // description and the caller's casing. - if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.Description); err != nil { + // description and the caller's casing. A statement without a + // description clause keeps the stored one: passing "" erased it, + // reported only as "Modified" (ako/mxcli#1218). + description := mr.Description + if s.Description != nil { + description = *s.Description + } + if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, description); err != nil { return mdlerrors.NewBackend("modify module role", err) } if !ctx.Quiet { @@ -76,7 +84,7 @@ func execCreateModuleRole(ctx *ExecContext, s *ast.CreateModuleRoleStmt) error { return mdlerrors.NewAlreadyExists("module role", s.Name.Module+"."+s.Name.Name) } - if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.Description); err != nil { + if err := ctx.Backend.AddModuleRole(ms.ID, s.Name.Name, s.DescriptionText()); err != nil { return mdlerrors.NewBackend("create module role", err) } diff --git a/mdl/executor/issue1218_module_role_description_test.go b/mdl/executor/issue1218_module_role_description_test.go new file mode 100644 index 000000000..767bb64d0 --- /dev/null +++ b/mdl/executor/issue1218_module_role_description_test.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/security" +) + +// ako/mxcli#1218: "A bare `create or modify module role M.R;` silently erases +// the role's description". The statement reported "Modified module role" and +// the role's Description was gone: the executor handed the statement's empty +// description to AddModuleRole, which overwrites. An omitted clause keeps what +// is stored, as an omitted user role property already did (ako/mxcli#707); a +// description clause with an empty string is how a script clears it. +// +// The statements are parsed with the real visitor, so the test covers the +// whole chain from "the statement does not say" to the value written. +func TestIssue1218_CreateOrModifyModuleRole_KeepsUnstatedDescription(t *testing.T) { + const stored = "Read the published OData services. Nothing else." + cases := []struct { + name string + src string + stored *security.ModuleRole // nil: the role does not exist yet + want string // the description AddModuleRole writes + }{ + {"bare statement keeps the stored description", + "create or modify module role Shop.ApiUser;", + &security.ModuleRole{Name: "ApiUser", Description: stored}, stored}, + {"a stated description replaces it", + "create or modify module role Shop.ApiUser description 'Calls the API';", + &security.ModuleRole{Name: "ApiUser", Description: stored}, "Calls the API"}, + {"an empty description clears it", + "create or modify module role Shop.ApiUser description '';", + &security.ModuleRole{Name: "ApiUser", Description: stored}, ""}, + {"a casing change keeps the stored description", + "create or modify module role Shop.apiUser;", + &security.ModuleRole{Name: "ApiUser", Description: stored}, stored}, + {"or replace reads as or modify and keeps it too", + "mdl 1;\ncreate or replace module role Shop.ApiUser;", + &security.ModuleRole{Name: "ApiUser", Description: stored}, stored}, + // Adopting a role mxcli auto-provisioned takes it over: its marker + // description is not the author's documentation, so it is not kept. + {"adopting an auto-provisioned role drops its marker", + "create module role Shop.ApiUser;", + &security.ModuleRole{Name: "ApiUser", Description: autoDocumentRoleDescription}, ""}, + {"a new role without a description has none", + "create or modify module role Shop.ApiUser;", + nil, ""}, + {"a new role takes the stated description", + "create module role Shop.ApiUser description 'Calls the API';", + nil, "Calls the API"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + mod := &model.Module{BaseElement: model.BaseElement{ID: "mod-shop"}, Name: "Shop"} + ms := &security.ModuleSecurity{BaseElement: model.BaseElement{ID: "ms-shop"}, ContainerID: mod.ID} + if tc.stored != nil { + ms.ModuleRoles = []*security.ModuleRole{tc.stored} + } + var written []string + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleSecurityFunc: func(model.ID) (*security.ModuleSecurity, error) { return ms, nil }, + AddModuleRoleFunc: func(_ model.ID, _ string, description string) error { + written = append(written, description) + return nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + + prog, errs := visitor.Build(tc.src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + stmt := findStmt[*ast.CreateModuleRoleStmt](t, prog, tc.src) + if err := execCreateModuleRole(ctx, stmt); err != nil { + t.Fatalf("exec: %v\n%s", err, buf.String()) + } + if len(written) != 1 { + t.Fatalf("AddModuleRole called %d time(s), want 1\n%s", len(written), buf.String()) + } + if written[0] != tc.want { + t.Errorf("description written = %q, want %q\n%s", written[0], tc.want, buf.String()) + } + }) + } +} diff --git a/mdl/executor/issue707_describe_roundtrip_test.go b/mdl/executor/issue707_describe_roundtrip_test.go index 923c7868a..296dcd9fb 100644 --- a/mdl/executor/issue707_describe_roundtrip_test.go +++ b/mdl/executor/issue707_describe_roundtrip_test.go @@ -114,8 +114,8 @@ func TestIssue707_ModuleRoleDescriptionReparses(t *testing.T) { out := buf.String() stmt := findStmt[*ast.CreateModuleRoleStmt](t, reparse(t, out), out) - if stmt.Description != "Can't delete orders" { - t.Errorf("description: stored %q, re-exec stores %q\n%s", "Can't delete orders", stmt.Description, out) + if stmt.DescriptionText() != "Can't delete orders" { + t.Errorf("description: stored %q, re-exec stores %q\n%s", "Can't delete orders", stmt.DescriptionText(), out) } } diff --git a/mdl/visitor/module_role_create_statement_test.go b/mdl/visitor/module_role_create_statement_test.go index abdc78eab..6aaaf4296 100644 --- a/mdl/visitor/module_role_create_statement_test.go +++ b/mdl/visitor/module_role_create_statement_test.go @@ -35,7 +35,7 @@ func TestCreateModuleRole_IsACreateStatementKind(t *testing.T) { if !ok { t.Fatalf("%s: got %T", tc.src, prog.Statements[0]) } - if s.Name.String() != "M.Admin" || s.CreateOrModify != tc.orModify || s.Description != tc.desc { + if s.Name.String() != "M.Admin" || s.CreateOrModify != tc.orModify || s.DescriptionText() != tc.desc { t.Errorf("%s: got %+v", tc.src, s) } if got := len(prog.Deprecations) > 0; got != tc.deprecated { diff --git a/mdl/visitor/visitor_security.go b/mdl/visitor/visitor_security.go index 0defad6da..3f00edd35 100644 --- a/mdl/visitor/visitor_security.go +++ b/mdl/visitor/visitor_security.go @@ -27,7 +27,10 @@ func (b *Builder) ExitCreateModuleRoleStatement(ctx *parser.CreateModuleRoleStat } if ctx.DESCRIPTION() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { - stmt.Description = unquoteStringLit(sl) + // A pointer, so `description ''` (clear it) and no clause at all + // (keep what is stored) stay different statements (ako/mxcli#1218). + d := unquoteStringLit(sl) + stmt.Description = &d } } b.statements = append(b.statements, stmt) diff --git a/mdl/visitor/visitor_security_or_modify_test.go b/mdl/visitor/visitor_security_or_modify_test.go index 22521e404..f38cbed14 100644 --- a/mdl/visitor/visitor_security_or_modify_test.go +++ b/mdl/visitor/visitor_security_or_modify_test.go @@ -42,8 +42,8 @@ func TestCreateModuleRole_OrModify(t *testing.T) { if got.Name.Name != tc.wantName { t.Errorf("Name = %q, want %q", got.Name.Name, tc.wantName) } - if got.Description != tc.desc { - t.Errorf("Description = %q, want %q", got.Description, tc.desc) + if got.DescriptionText() != tc.desc { + t.Errorf("Description = %q, want %q", got.DescriptionText(), tc.desc) } }) } diff --git a/mdl/visitor/visitor_security_test.go b/mdl/visitor/visitor_security_test.go index 440717b87..00e783c1a 100644 --- a/mdl/visitor/visitor_security_test.go +++ b/mdl/visitor/visitor_security_test.go @@ -28,8 +28,25 @@ func TestCreateModuleRole(t *testing.T) { if stmt.Name.Module != "MyModule" || stmt.Name.Name != "Editor" { t.Errorf("Expected MyModule.Editor, got %s.%s", stmt.Name.Module, stmt.Name.Name) } - if stmt.Description != "Can edit content" { - t.Errorf("Expected description 'Can edit content', got %q", stmt.Description) + if stmt.DescriptionText() != "Can edit content" { + t.Errorf("Expected description 'Can edit content', got %q", stmt.DescriptionText()) + } +} + +// A description clause with an empty string states an empty description, which +// clears a stored one under `create or modify`; no clause states none, which +// keeps it (ako/mxcli#1218). +func TestCreateModuleRole_EmptyDescriptionIsStated(t *testing.T) { + prog, errs := Build(`create or modify module role MyModule.Viewer description '';`) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.CreateModuleRoleStmt) + if !ok { + t.Fatalf("Expected CreateModuleRoleStmt, got %T", prog.Statements[0]) + } + if stmt.Description == nil || *stmt.Description != "" { + t.Errorf("Description = %v, want a stated empty string", stmt.Description) } } @@ -46,8 +63,8 @@ func TestCreateModuleRole_NoDescription(t *testing.T) { if !ok { t.Fatalf("Expected CreateModuleRoleStmt, got %T", prog.Statements[0]) } - if stmt.Description != "" { - t.Errorf("Expected empty description, got %q", stmt.Description) + if stmt.Description != nil { + t.Errorf("Expected no description clause, got %q", *stmt.Description) } } From 7f0d71e39c6e5bfd3b3a09dc529412e872805c8a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:28:53 +0000 Subject: [PATCH 04/44] fix(modelsdk): declare AggregateFunction on Mendix 9, floor Reduce's fold at 10.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom (ako/mxcli#1163, 02-microflow-examples on 9.24.30): every count/sum/average/minimum/maximum in a microflow was refused with "this Mendix 9.24.30 project cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". Cause: not an older storage shape. The vendored generated version data gave aggregateFunction the row of reduceReturnDataType ({Introduced 10.0.0, Required}) and had no row for reduceReturnDataType, so the property floor refused the key every aggregate carries and let the fold's return type through unrefused. Evidence: mendixmodelsdk 4.117.0 has aggregateFunction with no version (its default even changes at 9.24.0) and reduceReturnDataType introduced 10.0.0. Studio Pro 9.24.0 stores AggregateFunction on 10 of 10 aggregates in Workflow Commons 2.6.0. A floor-off build's aggregate survives `mx convert` 9.24.30 with every key. Fix: correct the Microflows$AggregateAction row by hand (gen cannot be regenerated here) and regenerate the copy in modelsdk/version. Measured: bug-test 1163-aggregate-on-mendix9.mdl on a fresh 9.24.30 app: exec writes all five aggregates, mx check 0 errors, mx convert strips nothing. Before, the first was refused. Control: the new tests fail on the unfixed table with the reported refusal. 02-microflow-examples on 9.24 now stops at the expression aggregate (`sum … of …`, UseExpression and Expression, Mendix 10.0+), which is correct. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...ggregate-function-floored-at-mendix10.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 5 +- .../bug-tests/1163-aggregate-on-mendix9.mdl | 36 ++++++++ modelsdk/gen/microflows/version.go | 8 +- .../mpr/writer_version_shape_1163_test.go | 83 +++++++++++++++++++ modelsdk/version/metamodel_versions_gen.go | 8 +- 7 files changed, 138 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-aggregate-function-floored-at-mendix10.json create mode 100644 mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl create mode 100644 modelsdk/mpr/writer_version_shape_1163_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-aggregate-function-floored-at-mendix10.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-aggregate-function-floored-at-mendix10.json new file mode 100644 index 000000000..bac85dc0a --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-aggregate-function-floored-at-mendix10.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x every count/sum/average/minimum/maximum in a microflow was refused: `this Mendix 9.24.30 project cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)` (02-microflow-examples on 9.24)", "cause": "The vendored generated version data (modelsdk/gen/microflows/version.go) gave aggregateFunction the row of reduceReturnDataType ({Introduced 10.0.0, Required}) and had no row for reduceReturnDataType, so the floor refused the one key every aggregate has and let the fold's return type through unrefused", "file": "`modelsdk/gen/microflows/version.go` (Microflows$AggregateAction, fixed by hand), `modelsdk/version/metamodel_versions_gen.go` (go generate ./modelsdk/version)", "insight": "**A refusal can be the floor's data being wrong, not the writer's shape.** The prompt to look for an 'older shape' was a guess; evidence settled it in two steps: mendixmodelsdk 4.117.0 has `aggregateFunction: {}` (no version) and even sets its default differently either side of 9.24.0, and Studio Pro-authored 9.24.0 bytes (Workflow Commons 2.6.0 from the Marketplace, `mxcli marketplace download 117066 --version 2.6.0`, project.mpr inside the .mpk) store `AggregateFunction: \"Count\"` on 10 of 10 aggregates. A one-off script comparing every SDK StructureVersionInfo `introduced` with modelsdk/gen/*/version.go found only 4 mismatches in the whole table (this one; CustomWidgets$WidgetReturnType.type 9.20 vs none; Forms$GroupBox.caption 8.10 vs none; Workflows$CallMicroflowTask.boundaryEvents), so the generator's misattribution is rare, not systematic. **The cheapest 'which keys does version X declare' oracle is a floor-off build + `mx convert -p` on that version**: build mxcli with StripUndeclaredProperties stubbed, exec, dump units, convert, dump again, diff key sets per $Type — 9.24's convert kept all of AggregateAction's keys. Measured: 5 aggregates on a fresh 9.24.30 app → exec ok, mx check 0 errors, mx convert strips nothing (before: refused). Tests: TestNormalize_AggregateFunctionIsDeclaredOnMendix9, TestNormalize_ReduceFoldIsRefusedBelowMendix10; mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl", "refs": ["#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index a947aaac4..d0905b71c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. - **A button's workflow action survives a rewrite** (ako/mxcli#832) — `describe` printed no action for Show user task page, Show workflow admin page or Call workflow, so re-executing a page turned them into no action (on ako/TestApp's TaskInbox and TaskDashboard, 2 of 6 user-task buttons each). They are now `open user task [with (AssignOnOpen, OpenWhenAssigned)]`, `open workflow with (DefaultPage: M.Page)` and `call workflow M.WF [with (…)]`. `complete task` takes `with (ClosePage, Commit)` instead of always writing both true. `open workflow` without a page is CE1834 and is refused. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 67099ac20..05ee85520 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -186,11 +186,12 @@ Some page properties kept their meaning and changed how they are stored. mxcli b | A text box's, text area's or date picker's placeholder | 10.11 | `Placeholder` text, instead of a `PlaceholderTemplate` | | Any data source's `ForceFullObjects` | 10.8 | Not stored | -What an older shape cannot hold is refused, naming the widget, rather than dropped: +What an older shape cannot hold is refused, naming the element, rather than dropped: - a Spacing, colour-picker or toggle-button-group design property below 10.2; - conditional data view editability below 10.7; - a placeholder with parameters below 10.11; -- a data view fed by another widget below 10.0. +- a data view fed by another widget below 10.0; +- an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl b/mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl new file mode 100644 index 000000000..d13e79e3a --- /dev/null +++ b/mdl-examples/bug-tests/1163-aggregate-on-mendix9.mdl @@ -0,0 +1,36 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x every count/sum/average/minimum/maximum in a +-- microflow was refused: +-- +-- this Mendix 9.24.30 project cannot store +-- Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+) +-- +-- The property floor came from the vendored generated version data, which had +-- given AggregateFunction the row of ReduceReturnDataType (the fold Reduce +-- added in 10.0) and left that one out. Studio Pro 9.24.0 stores +-- AggregateFunction on every aggregate (Workflow Commons 2.6.0), `mx convert` +-- 9.24.30 keeps it, and mendixmodelsdk gives it no version. +-- +-- Run it on a fresh 9.24.30 app (`mxcli new … --version 9.24.30.51827`): exec +-- writes all five, and `mxcli docker check` reports the blank app's error +-- count. An aggregate over an expression (`sum $L of …`) and `reduce`, `all` +-- and `any` are Mendix 10.0 and are still refused below it. + +create module Probe1163Agg; + +create persistent entity Probe1163Agg.Line ( + Amount: Decimal +); + +create microflow Probe1163Agg.Totals ( + $Lines: list of Probe1163Agg.Line +) +returns decimal as $Sum +begin + $Count = count $Lines; + $Sum = sum $Lines by Amount; + $Average = average $Lines by Amount; + $Min = minimum $Lines by Amount; + $Max = maximum $Lines by Amount; + return $Sum; +end; diff --git a/modelsdk/gen/microflows/version.go b/modelsdk/gen/microflows/version.go index 01eea8f03..dcd1c8162 100644 --- a/modelsdk/gen/microflows/version.go +++ b/modelsdk/gen/microflows/version.go @@ -45,9 +45,15 @@ var VersionInfos = map[string]version.TypeVersionInfo{ }, "Microflows$AggregateAction": { Properties: map[string]version.PropertyVersionInfo{ - "aggregateFunction": {Introduced: "10.0.0", Required: true}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave + // aggregateFunction the row of reduceReturnDataType, which it then + // left out. mendixmodelsdk 4.117.0 has aggregateFunction with no + // version and reduceReturnDataType introduced 10.0.0 and required; + // Studio Pro 9.24 stores AggregateFunction (Workflow Commons 2.6.0). + "aggregateFunction": {}, "expression": {Introduced: "10.0.0"}, "reduceInitialValueExpression": {Introduced: "10.0.0"}, + "reduceReturnDataType": {Introduced: "10.0.0", Required: true}, "useExpression": {Introduced: "10.0.0"}, }, }, diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go new file mode 100644 index 000000000..d62ab14de --- /dev/null +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/mpr/version" +) + +// ako/mxcli#1163: what the 9.24 doctype run found mxcli writing that Mendix 9 +// stores differently, or not at all. Each case is the document as mxcli builds +// it, normalized for the project's version the way every unit write is. + +// aggregateMicroflow is a microflow holding one Aggregate list activity as the +// writer builds it: Count/Sum/… carry no fold; Reduce, All and Any carry +// ReduceInitialValueExpression and ReduceReturnDataType. +func aggregateMicroflow(function string) bson.D { + action := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Microflows$AggregateAction"}, + {Key: "AggregateFunction", Value: function}, + {Key: "AggregateVariableName", Value: "Items"}, + {Key: "Attribute", Value: "M.Item.Price"}, + {Key: "ErrorHandlingType", Value: "Rollback"}} + switch function { + case "Reduce", "All", "Any": + initial := "" + if function == "Reduce" { + initial = "0" + } + action = append(action, + bson.E{Key: "ReduceInitialValueExpression", Value: initial}, + bson.E{Key: "ReduceReturnDataType", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "DataTypes$BooleanType"}}}) + } + action = append(action, bson.E{Key: "VariableName", Value: "Total"}) + activity := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Microflows$ActionActivity"}, {Key: "Action", Value: action}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Microflows$Microflow"}, {Key: "Name", Value: "Agg"}, + {Key: "ObjectCollection", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "Objects", Value: bson.A{int32(3), activity}}}}} +} + +// Microflows$AggregateAction.AggregateFunction is declared on every version +// mxcli writes. The generated version data floored it at 10.0.0, so every +// count/sum/average/minimum/maximum on 9.x was refused ("cannot store +// Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)"). Studio Pro +// 9.24.0 stores it (Workflow Commons 2.6.0: 10 of 10 aggregates, `Count`), `mx +// convert` 9.24.30 keeps it, and mendixmodelsdk gives it no version at all; the +// 10.0.0 floor belongs to ReduceReturnDataType, the fold Reduce added. +func TestNormalize_AggregateFunctionIsDeclaredOnMendix9(t *testing.T) { + for _, v := range []*version.ProjectVersion{pv(9, 0, 5), pv(9, 24, 30), pv(10, 0, 0), pv(11, 13, 0)} { + d, err := normalizeShape(t, v, aggregateMicroflow("Sum")) + if err != nil { + t.Errorf("%d.%d: a sum aggregate is refused: %v", v.MajorVersion, v.MinorVersion, err) + continue + } + if f, _ := get(find(d, "Microflows$AggregateAction"), "AggregateFunction"); f != "Sum" { + t.Errorf("%d.%d: AggregateFunction = %v, want Sum", v.MajorVersion, v.MinorVersion, f) + } + } +} + +// The fold is what Mendix 10.0 added (mendixmodelsdk: ReduceReturnDataType +// introduced 10.0.0). `all` carries an empty initial value, so the fold's +// return type is the one key that says it: on 9.24 it is refused, naming it, +// rather than written into a document Studio Pro 9 cannot resolve. Before, the +// generated data had no row for it and it passed through. +func TestNormalize_ReduceFoldIsRefusedBelowMendix10(t *testing.T) { + for _, fn := range []string{"Reduce", "All"} { + if _, err := normalizeShape(t, pv(9, 24, 30), aggregateMicroflow(fn)); err == nil || + !strings.Contains(err.Error(), "Microflows$AggregateAction.ReduceReturnDataType (Mendix 10.0.0+)") { + t.Errorf("%s on 9.24 was not refused naming its fold's return type: %v", fn, err) + } + d, err := normalizeShape(t, pv(10, 0, 0), aggregateMicroflow(fn)) + if err != nil { + t.Fatalf("%s on 10.0 is refused: %v", fn, err) + } + if _, ok := get(find(d, "Microflows$AggregateAction"), "ReduceReturnDataType"); !ok { + t.Errorf("%s on 10.0 lost ReduceReturnDataType", fn) + } + } +} diff --git a/modelsdk/version/metamodel_versions_gen.go b/modelsdk/version/metamodel_versions_gen.go index d5e02fa57..37591339c 100644 --- a/modelsdk/version/metamodel_versions_gen.go +++ b/modelsdk/version/metamodel_versions_gen.go @@ -1241,9 +1241,15 @@ var metamodelVersions = []map[string]TypeVersionInfo{ }, "Microflows$AggregateAction": { Properties: map[string]PropertyVersionInfo{ - "aggregateFunction": {Introduced: "10.0.0", Required: true}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave + // aggregateFunction the row of reduceReturnDataType, which it then + // left out. mendixmodelsdk 4.117.0 has aggregateFunction with no + // version and reduceReturnDataType introduced 10.0.0 and required; + // Studio Pro 9.24 stores AggregateFunction (Workflow Commons 2.6.0). + "aggregateFunction": {}, "expression": {Introduced: "10.0.0"}, "reduceInitialValueExpression": {Introduced: "10.0.0"}, + "reduceReturnDataType": {Introduced: "10.0.0", Required: true}, "useExpression": {Introduced: "10.0.0"}, }, }, From 57cb3179ade693af0a1c4ea78b68ab56ba6d38b0 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:41:27 +0000 Subject: [PATCH 05/44] fix(canon): write no workflow PersistentId below Mendix 10.21 Symptom (ako/mxcli#1163, 25-rename on 9.24.30): every workflow was refused with "this Mendix 9.24.30 project cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". The same held on 10.0-10.20. Cause: the workflow writer, and the ALTER WORKFLOW mutator for the activities it inserts, mint a fresh PersistentId on the workflow and on every activity, outcome and boundary event. mendixmodelsdk introduces PersistentId in 10.21.0 on all four (Workflow, and the abstract WorkflowActivity, Outcome and BoundaryEvent). The floor refused the workflow's own. It never saw the others, which are declared on supertypes it does not look up. `mx convert` 9.24.30 strips the PersistentId from all 6 element types a floor-off build wrote. Fix: canon.LowerToProjectShape drops PersistentId from every Workflows$ element below 10.21. It runs on every unit write before the floor, so CREATE, CREATE OR MODIFY and wfmutator's raw inserts are all covered. The ID is dropped rather than refused because the writer mints it; no script states it, and Studio Pro 9.24 stores none. Measured: bug-test 1163-workflow-on-mendix9.mdl on a fresh 9.24.30 app: exec ok, mx check 0 errors, `mx convert -p` strips no key. 25-rename on 9.24: FAIL (the refusal) -> PASS. Controls: 25-rename and 24b-workflow-mendix10 PASS on 10.24.24 and 11.13.0. The unit test fails on the unfixed code with the reported refusal. Not fixed: a parallel split writes Workflows$EndOfParallelSplitPathActivity, a 10.21 type, and 9.24 then does not load the project. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-28-workflow-persistentid-below-10-21.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 5 +- .../bug-tests/1163-workflow-on-mendix9.mdl | 55 +++++++++++++ modelsdk/canon/versionshape.go | 27 +++++- .../mpr/writer_version_shape_1163_test.go | 82 +++++++++++++++++++ 6 files changed, 167 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-workflow-persistentid-below-10-21.json create mode 100644 mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-workflow-persistentid-below-10-21.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-workflow-persistentid-below-10-21.json new file mode 100644 index 000000000..52ce2c428 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-workflow-persistentid-below-10-21.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x and 10.0–10.20 every workflow was refused: `this Mendix 9.24.30 project cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)` (25-rename on 9.24)", "cause": "The workflow writer (and the ALTER WORKFLOW mutator, for the activities it inserts) mints a fresh PersistentId on the workflow and on every activity, outcome and boundary event. PersistentId is 10.21.0 on all four (Workflow, and the abstract WorkflowActivity, Outcome, BoundaryEvent). The floor refused the workflow's own and never saw the rest, because it does not look up abstract supertypes", "file": "`modelsdk/canon/versionshape.go` (LowerToProjectShape: dropPersistentID on every Workflows$ element below 10.21)", "insight": "**An identity the writer mints is not a value the statement said**, so below its version it is dropped, not refused (unlike ForceFullObjects, which a script can set). Lower it in canon.LowerToProjectShape rather than encodeWorkflow: wfmutator's raw inserts (mutator.go: PersistentId on inserted activities, outcomes, boundary events) never pass encodeWorkflow, but every write passes normalizePropertySetsFor. **Test with a non-zero GUID**: a zero GUID is an empty value the floor silently drops, so the first draft of the test passed on the unfixed code. **The floor-off build + `mx convert -p` 9.24 key diff** showed the whole picture at once: PersistentId on 6 element types, plus the empty keys the floor already handles. The same run found the next gap: a `parallel split` writes Workflows$EndOfParallelSplitPathActivity (a 10.21 type) and 9.24 does not load the project (TypeCacheUnknownTypeException) — not fixed here. Measured: bug-test on a fresh 9.24.30 app → exec ok, mx check 0 errors, mx convert strips no key (adds WorkflowOnStateChangeEvent/UsertaskOnStateChangeEvent null). 25-rename 9.24 FAIL→PASS; 25-rename and 24b on 10.24.24 and 11.13 PASS. Tests: TestNormalize_WorkflowPersistentIDsBelowMendix10_21; mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl", "refs": ["#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index d0905b71c..5a03ede06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 05ee85520..3f773a76f 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -174,9 +174,9 @@ What *is* version-specific is the optional/default-value half: `PageParameter.Is Atlas UI v3 (Mendix 11.0+) introduced new design properties like "Card style" and "Disable row wrap" for containers. These don't exist in the Atlas v2 theme bundled with 10.x, causing CE6083. -### Page Properties Stored in an Older Shape (below 10.2, 10.7, 10.11) +### Properties Stored in an Older Shape -Some page properties kept their meaning and changed how they are stored. mxcli builds the current shape and, on every write, rewrites it into the shape the project's version stores. `describe` reads both shapes ([ako/mxcli#1209](https://github.com/ako/mxcli/issues/1209)). The older shapes were measured on Studio Pro-authored 9.24.30 pages: +Some properties kept their meaning and changed how they are stored. mxcli builds the current shape and, on every write, rewrites it into the shape the project's version stores. `describe` reads both shapes ([ako/mxcli#1209](https://github.com/ako/mxcli/issues/1209), [ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). The older shapes were measured on Studio Pro-authored 9.24 documents and with `mx convert` 9.24.30: | Property | Below | Stored as | |---|---|---| @@ -185,6 +185,7 @@ Some page properties kept their meaning and changed how they are stored. mxcli b | A data view's editability | 10.7 | `Editable: true\|false`, instead of `Editability` | | A text box's, text area's or date picker's placeholder | 10.11 | `Placeholder` text, instead of a `PlaceholderTemplate` | | Any data source's `ForceFullObjects` | 10.8 | Not stored | +| The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | What an older shape cannot hold is refused, naming the element, rather than dropped: - a Spacing, colour-picker or toggle-button-group design property below 10.2; diff --git a/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl b/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl new file mode 100644 index 000000000..ecea673ad --- /dev/null +++ b/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl @@ -0,0 +1,55 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x and on 10.0–10.20 every workflow was refused: +-- +-- this Mendix 9.24.30 project cannot store +-- Workflows$Workflow.PersistentId (Mendix 10.21.0+) +-- +-- Workflows exist from 9.0; PersistentId arrived in 10.21.0, on the workflow +-- and on every activity, outcome and boundary event (mendixmodelsdk). The +-- activities' and outcomes' were declared on abstract supertypes the property +-- floor does not look up, so they were never refused, and `mx convert` 9.24.30 +-- strips every one. A PersistentId is minted by the writer, never stated by a +-- script, so below 10.21 none is written: for CREATE, CREATE OR MODIFY and +-- the activities ALTER WORKFLOW inserts on its own. +-- +-- Run it on a fresh 9.24.30 app (`mxcli new … --version 9.24.30.51827`): exec, +-- then `mxcli docker check` reports the blank app's error count, and `mx +-- convert -p` strips no PersistentId. +-- +-- Not covered: a `parallel split`. Its paths end in a +-- Workflows$EndOfParallelSplitPathActivity, a type Mendix introduced in 10.21, +-- so below 10.21 the project does not load (TypeCacheUnknownTypeException). + +create module Probe1163Wf; + +create persistent entity Probe1163Wf.Request ( + Title: String(100), + Amount: Integer +); + +create microflow Probe1163Wf.ACT_Validate ($WorkflowContext: Probe1163Wf.Request) +begin + log info node 'Probe1163Wf' 'validate'; +end; + +create microflow Probe1163Wf.ACT_Notify ($WorkflowContext: Probe1163Wf.Request) +begin + log info node 'Probe1163Wf' 'notify'; +end; + +create workflow Probe1163Wf.WF_Request + parameter $WorkflowContext: Probe1163Wf.Request + display 'Request' +begin + call microflow Probe1163Wf.ACT_Validate; + decision $WorkflowContext/Amount > 1000 + outcomes + true -> { call microflow Probe1163Wf.ACT_Notify; } + false -> { end workflow; } + ; +end workflow; + +-- ALTER serializes the inserted activity on its own, outside CREATE's encoder. +alter workflow Probe1163Wf.WF_Request { + insert after ACT_Validate { call microflow Probe1163Wf.ACT_Notify; } +}; diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index fb1efebde..31be06cae 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -4,6 +4,7 @@ package canon import ( "fmt" + "strings" "go.mongodb.org/mongo-driver/v2/bson" @@ -37,6 +38,10 @@ import ( // CE6050 "The selected entity must be configured with a page parameter". // - ForceFullObjects, declared on the abstract Forms$DataSource from 10.8.0: // every data source carried it, and the floor never saw it. +// - Workflows$ PersistentId, from 10.21.0 on the workflow and on the abstract +// WorkflowActivity, Outcome and BoundaryEvent (ako/mxcli#1163). Workflows +// exist from 9.0; `mx convert` 9.24.30 strips every one. The workflow's +// own was refused; the others slipped past the floor. // // It runs on every unit write, before the floor, so no writer needs to know // these shapes. What the older shape cannot hold — a design property value @@ -53,8 +58,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { textPlaceholder: !PropertyDeclared("Forms$AttributeWidgetWithPlaceholder", "PlaceholderTemplate", *pv), namedParameter: !PropertyDeclared("Forms$EntityPathSource", "SourceVariable", *pv), noFullObjects: !PropertyDeclared("Forms$DataSource", "ForceFullObjects", *pv), + noPersistentID: !PropertyDeclared("Workflows$Workflow", "PersistentId", *pv), } - if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects { + if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && + !l.noPersistentID { return contents, nil } var doc bson.D @@ -74,7 +81,7 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { type shapeLowering struct { pv version.Version flatDesignProps, boolEditable, textPlaceholder bool - namedParameter, noFullObjects bool + namedParameter, noFullObjects, noPersistentID bool changed bool } @@ -112,6 +119,9 @@ func (l *shapeLowering) walk(v any) (any, error) { return nil, err } } + if l.noPersistentID && strings.HasPrefix(typ, "Workflows$") { + x = l.dropPersistentID(x) + } for i := range x { if x[i].Value, err = l.walk(x[i].Value); err != nil { return nil, err @@ -261,6 +271,19 @@ func (l *shapeLowering) dropForceFullObjects(d bson.D, typ string) (bson.D, erro return append(append(bson.D{}, d[:i]...), d[i+1:]...), nil } +// dropPersistentID removes a workflow element's PersistentId below 10.21, where +// none declares it. Whatever it holds is dropped, not refused: a PersistentId is +// minted by the writer (a fresh GUID on every build), never stated by a script, +// and a workflow Studio Pro 9.24 stores has none. +func (l *shapeLowering) dropPersistentID(d bson.D) bson.D { + i := shapeIndex(d, "PersistentId") + if i < 0 { + return d + } + l.changed = true + return append(append(bson.D{}, d[:i]...), d[i+1:]...) +} + // listItems returns the elements of a stored list, without its type marker. func listItems(v any) bson.A { a, _ := v.(bson.A) diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index d62ab14de..f3ae39ac5 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -81,3 +81,85 @@ func TestNormalize_ReduceFoldIsRefusedBelowMendix10(t *testing.T) { } } } + +// pid is a fresh PersistentId: a non-zero GUID, as the writer mints one (a zero +// GUID is an empty value, which the floor would drop rather than refuse). +func pid() bson.Binary { + pidSeq++ + return bson.Binary{Subtype: 0, Data: []byte{pidSeq, 0x11, 2, 3, 4, 5, 0x46, 7, 0x88, 9, 10, 11, 12, 13, 14, 15}} +} + +var pidSeq byte + +// workflowWithPersistentIDs is a workflow as mxcli builds it: a fresh +// PersistentId on the workflow, on every activity and on every outcome. +func workflowWithPersistentIDs() bson.D { + outcome := func(value bool) bson.D { + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$BooleanConditionOutcome"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: bson.A{int32(3)}}}}, + {Key: "PersistentId", Value: pid()}, {Key: "Value", Value: value}} + } + split := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$ExclusiveSplitActivity"}, + {Key: "Caption", Value: "Active?"}, {Key: "Expression", Value: "$WorkflowContext/Active"}, {Key: "Name", Value: "split"}, + {Key: "Outcomes", Value: bson.A{int32(3), outcome(true), outcome(false)}}, + {Key: "PersistentId", Value: pid()}} + call := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$CallMicroflowTask"}, + {Key: "BoundaryEvents", Value: bson.A{int32(2)}}, {Key: "Caption", Value: "Validate"}, + {Key: "Microflow", Value: "M.Validate"}, {Key: "Name", Value: "call"}, {Key: "PersistentId", Value: pid()}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), call, split}}}}, + {Key: "Name", Value: "WF"}, + {Key: "PersistentId", Value: pid()}, + {Key: "Title", Value: "WF"}} +} + +// persistentIDHolders returns the $Type of every element in v carrying a +// PersistentId. +func persistentIDHolders(v any) []string { + var out []string + switch x := v.(type) { + case bson.D: + typ, _ := get(x, "$Type") + for _, e := range x { + if e.Key == "PersistentId" { + out = append(out, typ.(string)) + } + out = append(out, persistentIDHolders(e.Value)...) + } + case bson.A: + for _, el := range x { + out = append(out, persistentIDHolders(el)...) + } + } + return out +} + +// Workflows exist from 9.0, PersistentId from 10.21.0 — on the workflow and on +// WorkflowActivity, Outcome and BoundaryEvent (mendixmodelsdk). Below that a +// workflow was refused ("cannot store Workflows$Workflow.PersistentId (Mendix +// 10.21.0+)"), and the activities' and outcomes' were never seen by the floor, +// which does not look up the abstract supertypes that declare them: `mx +// convert` 9.24.30 strips all of them. A PersistentId is minted by the writer, +// never stated by a script, so below 10.21 it is not written. +func TestNormalize_WorkflowPersistentIDsBelowMendix10_21(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + want int + }{ + {pv(9, 24, 30), 0}, + {pv(10, 20, 0), 0}, + {pv(10, 21, 0), 5}, + {pv(10, 24, 24), 5}, + {pv(11, 13, 0), 5}, // control + } { + d, err := normalizeShape(t, tt.v, workflowWithPersistentIDs()) + if err != nil { + t.Errorf("%d.%d: a workflow is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + if got := persistentIDHolders(d); len(got) != tt.want { + t.Errorf("%d.%d: PersistentId on %d elements %v, want %d", tt.v.MajorVersion, tt.v.MinorVersion, len(got), got, tt.want) + } + } +} From 2c7876b6b9a461a1c773675a5abef512148bb355 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:43:08 +0000 Subject: [PATCH 06/44] fix(canon): complete a workflow's state-change handlers on Mendix 9.24 Symptom (found while measuring ako/mxcli#1163): with PersistentId lowered, `mx convert -p` 9.24.30 strips nothing from an mxcli-written workflow but adds WorkflowOnStateChangeEvent: null and UsertaskOnStateChangeEvent: null. Studio Pro's next save then differs in property names, which Mendix's merge engine refuses to compare (#1373). Cause: the two defaults (mendixmodelsdk: introduced 9.12.0, removed 11.0.0) were measured on 10.24.24 only, and an entry is added only from the lowest version it was measured on. Fix: lower their measured floor to 9.24.30, where `mx convert` adds both with the same null value. Measured: bug-test 1163-workflow-on-mendix9.mdl on a fresh 9.24.30 app: mx check 0 errors, and `mx convert -p` now neither strips nor adds a key on the workflow. Before, it added these two. The test fails on the previous floor (present false on 9.24). The 11.13 case stays absent as the control. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- CHANGELOG.md | 1 + .../canon/studiopro_property_defaults.json | 4 +-- .../mpr/writer_version_shape_1163_test.go | 26 +++++++++++++++++++ 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a03ede06..bcd5486ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). - **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. diff --git a/modelsdk/canon/studiopro_property_defaults.json b/modelsdk/canon/studiopro_property_defaults.json index 5ceeade35..f05785e61 100644 --- a/modelsdk/canon/studiopro_property_defaults.json +++ b/modelsdk/canon/studiopro_property_defaults.json @@ -109,6 +109,6 @@ {"type":"Workflows$InterruptingTimerBoundaryEvent","key":"Annotation","since":"11.6.8","seen":"11.15.0","value":{"v":null}}, {"type":"Workflows$NonInterruptingTimerBoundaryEvent","key":"Annotation","since":"11.6.8","seen":"11.15.0","value":{"v":null}}, {"type":"Workflows$NonInterruptingTimerEventSubProcessStartActivity","key":"Recurrence","since":"11.15.0","seen":"11.15.0","value":{"v":null}}, - {"type":"Workflows$Workflow","key":"UsertaskOnStateChangeEvent","since":"10.24.24","until":"10.25.0","seen":"10.24.24","value":{"v":null}}, - {"type":"Workflows$Workflow","key":"WorkflowOnStateChangeEvent","since":"10.24.24","until":"10.25.0","seen":"10.24.24","value":{"v":null}} + {"type":"Workflows$Workflow","key":"UsertaskOnStateChangeEvent","since":"9.24.30","until":"10.25.0","seen":"10.24.24","value":{"v":null}}, + {"type":"Workflows$Workflow","key":"WorkflowOnStateChangeEvent","since":"9.24.30","until":"10.25.0","seen":"10.24.24","value":{"v":null}} ] diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index f3ae39ac5..a916b70ea 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -163,3 +163,29 @@ func TestNormalize_WorkflowPersistentIDsBelowMendix10_21(t *testing.T) { } } } + +// Studio Pro's property set for a workflow on 9.24 includes the two state-change +// event handlers Mendix removed in 11.0: `mx convert` 9.24.30 adds both as null +// to a workflow mxcli wrote, as 10.24.24 does — a property-name difference +// Mendix's merge engine refuses to compare (#1373). +func TestNormalize_WorkflowStateChangeEventsOnMendix9(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + want bool + }{ + {pv(9, 24, 30), true}, + {pv(10, 24, 24), true}, + {pv(11, 13, 0), false}, // removed in 11.0 + } { + d, err := normalizeShape(t, tt.v, workflowWithPersistentIDs()) + if err != nil { + t.Fatal(err) + } + wf := find(d, "Workflows$Workflow") + for _, k := range []string{"WorkflowOnStateChangeEvent", "UsertaskOnStateChangeEvent"} { + if v, ok := get(wf, k); ok != tt.want || v != nil { + t.Errorf("%d.%d: %s present %v (%v), want present %v and null", tt.v.MajorVersion, tt.v.MinorVersion, k, ok, v, tt.want) + } + } + } +} From 4df889ee9ce46ef1b3826ce535e3c456049c0781 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:51:21 +0000 Subject: [PATCH 07/44] fix(canon): write a constant override flat below Mendix 10.9 Symptom (ako/mxcli#1163, 14-project-settings on 9.24.30): `alter settings constant @M.C value '...' in configuration 'Default'` was refused with "this Mendix 9.24.30 project cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". The same held on 10.0-10.8. Cause: the settings overlay writes a new override as a Settings$SharedValue nested under SharedOrPrivateValue. From 10.9 that is the only shape Mendix reads. Below 10.9 an override is {$ID, $Type, ConstantId, Value} with a flat string (mendixmodelsdk: Value deleted and SharedOrPrivateValue introduced in 10.9.0). With the floor off, `mx convert` 9.24.30 turns the nested form into Value: "" and the override silently becomes empty. Fix: canon.LowerToProjectShape replaces SharedOrPrivateValue with the flat Value in its place, or fills the empty flat sibling the overlay leaves. A Settings$PrivateValue has no flat form, so it is refused, naming the constant. The reader already preferred a flat Value; a backend test now pins reading and updating a 9.x flat override. Measured: bug-test 1163-constant-override-on-mendix9.mdl on a fresh 9.24.30 app: describe settings shows 'broker:9092', mx check 0 errors, and mx convert keeps the value and changes no key in the settings unit. 14-project-settings on 9.24: FAIL (the refusal) -> PASS. Controls: 10.24.24 and 11.13.0 PASS. The unit tests fail on the unfixed code with the reported refusal. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-28-constant-override-flat-below-10-9.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1163-constant-override-on-mendix9.mdl | 30 ++++++ .../settings_mendix9_constant_value_test.go | 101 ++++++++++++++++++ mdl/settingsoverlay/settingsoverlay.go | 8 +- modelsdk/canon/versionshape.go | 38 ++++++- .../mpr/writer_version_shape_1163_test.go | 71 ++++++++++++ 8 files changed, 248 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-constant-override-flat-below-10-9.json create mode 100644 mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl create mode 100644 mdl/backend/modelsdk/settings_mendix9_constant_value_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-constant-override-flat-below-10-9.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-constant-override-flat-below-10-9.json new file mode 100644 index 000000000..27ca2e65a --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-constant-override-flat-below-10-9.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x and 10.0–10.8 a new constant override was refused: `alter settings constant @M.C value '…' in configuration 'Default'` → `this Mendix 9.24.30 project cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)` (14-project-settings on 9.24)", "cause": "settingsoverlay writes a NEW override in the nested shape (Settings$SharedValue under SharedOrPrivateValue), which is the only shape 10.9+ reads; below 10.9 the override is `{$ID, $Type, ConstantId, Value}` with a flat string. The overlay already kept an existing flat override flat, so only a new one failed", "file": "`modelsdk/canon/versionshape.go` (LowerToProjectShape: lowerConstantValue), `mdl/settingsoverlay/settingsoverlay.go` (comment)", "insight": "**`mx convert` on the older version is a silent-data-loss detector, not just a key oracle**: given the nested form, 9.24.30 kept the override but emptied it (`Value: \"\"`) and still reported success — an unlowered write that got past the floor would have shipped an empty override with mx check at 0 errors. Measure the VALUE after convert, not only the key set. A private value (Settings$PrivateValue, workstation-local) has no flat form and is refused naming the constant. The reader (settings_read.go constantValueOf) already preferred the flat Value, so no reader change was needed — TestProjectSettings_Mendix9FlatConstantValue pins it. Measured: bug-test on a fresh 9.24.30 app → describe settings prints 'broker:9092', mx check 0 errors, mx convert keeps the value and changes no key in the settings unit. 14-project-settings 9.24 FAIL→PASS; 10.24.24 and 11.13 PASS. Tests: TestNormalize_ConstantValueIsFlatBelowMendix10_9, TestNormalize_RefusesAPrivateConstantValueBelowMendix10_9; mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl", "refs": ["#1163", "#801"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index bcd5486ec..5437a9653 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. - **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). - **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 3f773a76f..6256f085e 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -186,12 +186,14 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | A text box's, text area's or date picker's placeholder | 10.11 | `Placeholder` text, instead of a `PlaceholderTemplate` | | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | +| A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | What an older shape cannot hold is refused, naming the element, rather than dropped: - a Spacing, colour-picker or toggle-button-group design property below 10.2; - conditional data view editability below 10.7; - a placeholder with parameters below 10.11; - a data view fed by another widget below 10.0; +- a private constant value in a configuration below 10.9; - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl b/mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl new file mode 100644 index 000000000..a0d8db719 --- /dev/null +++ b/mdl-examples/bug-tests/1163-constant-override-on-mendix9.mdl @@ -0,0 +1,30 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x and on 10.0–10.8 a new constant override in a +-- configuration was refused: +-- +-- this Mendix 9.24.30 project cannot store +-- Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+) +-- +-- Before 10.9 an override's value is a flat Value string on the +-- Settings$ConstantValue; from 10.9 it is a Settings$SharedValue nested under +-- SharedOrPrivateValue (mendixmodelsdk). `mx convert` 9.24.30 turns the nested +-- form into `{$ID, $Type, ConstantId, Value: ""}`: it does not know it, and the +-- override silently becomes empty. mxcli builds the nested form and lowers it +-- on the write. A private value (stored on the developer's workstation) has no +-- flat form and is refused below 10.9. +-- +-- Run it on a fresh 9.24.30 app (`mxcli new … --version 9.24.30.51827`): exec, +-- then `describe settings` shows the override as 'broker:9092', and `mxcli +-- docker check` reports the blank app's error count. + +create module Probe1163Const; + +create constant Probe1163Const.ServerUrl ( Type: string, DefaultValue: 'localhost:9092' ); + +-- A new override: built nested, stored flat. +alter settings constant @Probe1163Const.ServerUrl value 'kafka:9092' + in configuration 'Default'; + +-- An existing flat override is updated in place. +alter settings constant @Probe1163Const.ServerUrl value 'broker:9092' + in configuration 'Default'; diff --git a/mdl/backend/modelsdk/settings_mendix9_constant_value_test.go b/mdl/backend/modelsdk/settings_mendix9_constant_value_test.go new file mode 100644 index 000000000..b574c4221 --- /dev/null +++ b/mdl/backend/modelsdk/settings_mendix9_constant_value_test.go @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/bsonutil" + "github.com/mendixlabs/mxcli/mdl/settingsoverlay" + "github.com/mendixlabs/mxcli/modelsdk/bsoncompat" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" +) + +// seedFlatConstantValue gives the fixture's Default configuration one constant +// override in the shape Mendix stores below 10.9 (ako/mxcli#1163; `mx convert` +// 9.24.30): `{$ID, $Type, ConstantId, Value}`, the value a flat string. +func seedFlatConstantValue(t *testing.T, proj, constantID, value string) { + t.Helper() + r, err := mmpr.OpenWithOptions(proj, mmpr.OpenOptions{ReadOnly: false}) + if err != nil { + t.Fatalf("open fixture: %v", err) + } + defer r.Close() + refs, err := r.ListUnitsByType("Settings$ProjectSettings") + if err != nil || len(refs) != 1 { + t.Fatalf("ListUnitsByType(Settings$ProjectSettings) = %v, %v", refs, err) + } + raw, err := r.GetRawUnitBytes(refs[0].ID) + if err != nil { + t.Fatalf("GetRawUnitBytes: %v", err) + } + var doc bson.M + if err := bsoncompat.UnmarshalDocM(raw, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + seeded := false + for _, part := range settingsoverlay.ArrayElements(doc["Settings"]) { + if part["$Type"] != "Settings$ConfigurationSettings" { + continue + } + cfgs := settingsoverlay.ArrayElements(part["Configurations"]) + if len(cfgs) == 0 { + t.Fatalf("fixture has no configurations to seed") + } + cfgs[0]["ConstantValues"] = bson.A{int32(3), bson.D{ + {Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, + {Key: "$Type", Value: "Settings$ConstantValue"}, + {Key: "ConstantId", Value: constantID}, + {Key: "Value", Value: value}, + }} + seeded = true + } + if !seeded { + t.Fatalf("fixture has no Settings$ConfigurationSettings part") + } + contents, err := bson.Marshal(bsonutil.OrderStorageValue(doc)) + if err != nil { + t.Fatalf("marshal: %v", err) + } + if err := mmpr.NewWriterWithReader(r).UpdateRawUnit(refs[0].ID, contents); err != nil { + t.Fatalf("UpdateRawUnit: %v", err) + } +} + +// A Mendix 9 constant override is read from its flat Value, and updating it +// keeps that shape rather than introducing the nested one 9.x does not have. +func TestProjectSettings_Mendix9FlatConstantValue(t *testing.T) { + proj := copyFixture(t) + const id = "MyFirstModule.ServerUrl" + seedFlatConstantValue(t, proj, id, "kafka:9092") + + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + ps, err := b.GetProjectSettings() + if err != nil { + t.Fatalf("GetProjectSettings: %v", err) + } + cvs := ps.Configuration.Configurations[0].ConstantValues + if len(cvs) != 1 || cvs[0].ConstantId != id || cvs[0].Value != "kafka:9092" { + t.Fatalf("flat override read as %+v, want %s = kafka:9092", cvs, id) + } + + cvs[0].Value = "broker:9092" + if err := b.UpdateProjectSettings(ps); err != nil { + t.Fatalf("UpdateProjectSettings: %v", err) + } + if err := b.Disconnect(); err != nil { + t.Fatalf("disconnect: %v", err) + } + after := settingsoverlay.ArrayElements(readConfiguration(t, proj)["ConstantValues"]) + if len(after) != 1 || after[0]["Value"] != "broker:9092" { + t.Fatalf("flat override after update = %#v, want Value broker:9092", after) + } + if _, nested := after[0]["SharedOrPrivateValue"]; nested { + t.Errorf("updating a flat override added the nested shape: %#v", after[0]) + } +} diff --git a/mdl/settingsoverlay/settingsoverlay.go b/mdl/settingsoverlay/settingsoverlay.go index a11bfa375..1434bf917 100644 --- a/mdl/settingsoverlay/settingsoverlay.go +++ b/mdl/settingsoverlay/settingsoverlay.go @@ -343,9 +343,11 @@ func SetWebUISettings(ws *model.WebUISettings, raw map[string]any) map[string]an // ConstantValues rebuilds a configuration's ConstantValues list, updating each // override in the slot it is already stored in so its value shape survives. -// Studio Pro and mxbuild only read the nested SharedOrPrivateValue; a flat "Value" -// is a legacy shape mxcli's reader tolerates but the platform ignores, so writing -// every override flat made all constant overrides look empty in Studio Pro. +// From Mendix 10.9 Studio Pro and mxbuild only read the nested +// SharedOrPrivateValue, so writing every override flat made all constant +// overrides look empty in Studio Pro. Below 10.9 the flat "Value" is the only +// shape: a new override is still built nested here, and the write lowers it +// (canon.LowerToProjectShape, ako/mxcli#1163). func ConstantValues(cvs []*model.ConstantValue, raw any) bson.A { rawValues := ArrayElements(raw) byConstant := make(map[string]map[string]any, len(rawValues)) diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 31be06cae..399afe049 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -38,6 +38,10 @@ import ( // CE6050 "The selected entity must be configured with a page parameter". // - ForceFullObjects, declared on the abstract Forms$DataSource from 10.8.0: // every data source carried it, and the floor never saw it. +// - Settings$ConstantValue: a Settings$SharedValue under SharedOrPrivateValue +// from 10.9.0; before, the value as a flat Value string, `{$ID, $Type, +// ConstantId, Value}` (`mx convert` 9.24.30, which empties the nested form +// it does not know). // - Workflows$ PersistentId, from 10.21.0 on the workflow and on the abstract // WorkflowActivity, Outcome and BoundaryEvent (ako/mxcli#1163). Workflows // exist from 9.0; `mx convert` 9.24.30 strips every one. The workflow's @@ -59,9 +63,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { namedParameter: !PropertyDeclared("Forms$EntityPathSource", "SourceVariable", *pv), noFullObjects: !PropertyDeclared("Forms$DataSource", "ForceFullObjects", *pv), noPersistentID: !PropertyDeclared("Workflows$Workflow", "PersistentId", *pv), + flatConstant: !PropertyDeclared("Settings$ConstantValue", "SharedOrPrivateValue", *pv), } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID { + !l.noPersistentID && !l.flatConstant { return contents, nil } var doc bson.D @@ -82,6 +87,7 @@ type shapeLowering struct { pv version.Version flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool + flatConstant bool changed bool } @@ -113,6 +119,10 @@ func (l *shapeLowering) walk(v any) (any, error) { if err = l.lowerPlaceholder(x, typ); err != nil { return nil, err } + case l.flatConstant && typ == "Settings$ConstantValue": + if x, err = l.lowerConstantValue(x); err != nil { + return nil, err + } } if l.noFullObjects { if x, err = l.dropForceFullObjects(x, typ); err != nil { @@ -271,6 +281,32 @@ func (l *shapeLowering) dropForceFullObjects(d bson.D, typ string) (bson.D, erro return append(append(bson.D{}, d[:i]...), d[i+1:]...), nil } +// lowerConstantValue replaces a constant override's nested SharedOrPrivateValue +// with the flat Value older versions store, in its place. A flat Value already +// beside it (the overlay clears one to "" when it writes the nested form) takes +// the nested value. A private value — kept on the developer's workstation, which +// the flat form cannot say — is refused. +func (l *shapeLowering) lowerConstantValue(d bson.D) (bson.D, error) { + i := shapeIndex(d, "SharedOrPrivateValue") + if i < 0 { + return d, nil + } + inner, _ := d[i].Value.(bson.D) + if t, _ := shapeString(inner, "$Type"); t == "Settings$PrivateValue" { + constant, _ := shapeString(d, "ConstantId") + return nil, fmt.Errorf("constant %q has a private value in this configuration, which Mendix introduced in 10.9 "+ + "(this project is %s): a constant value there is shared, stored in the model", constant, l.pv) + } + value, _ := shapeString(inner, "Value") + l.changed = true + if j := shapeIndex(d, "Value"); j >= 0 { + d[j].Value = value + return append(append(bson.D{}, d[:i]...), d[i+1:]...), nil + } + d[i] = bson.E{Key: "Value", Value: value} + return d, nil +} + // dropPersistentID removes a workflow element's PersistentId below 10.21, where // none declares it. Whatever it holds is dropped, not refused: a PersistentId is // minted by the writer (a fresh GUID on every build), never stated by a script, diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index a916b70ea..eb00c2189 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -164,6 +164,77 @@ func TestNormalize_WorkflowPersistentIDsBelowMendix10_21(t *testing.T) { } } +// settingsWithConstantValue is the project settings unit as `alter settings +// constant … in configuration` writes a new override: the value nested in a +// Settings$SharedValue (or a Settings$PrivateValue, which holds none). flat is +// the empty flat Value the overlay leaves beside a nested one, "-" for none. +func settingsWithConstantValue(valueType, value, flat string) bson.D { + inner := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: valueType}} + if valueType == "Settings$SharedValue" { + inner = append(inner, bson.E{Key: "Value", Value: value}) + } + cv := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Settings$ConstantValue"}, + {Key: "ConstantId", Value: "M.ServerUrl"}, {Key: "SharedOrPrivateValue", Value: inner}} + if flat != "-" { + cv = append(cv, bson.E{Key: "Value", Value: flat}) + } + config := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Settings$ServerConfiguration"}, + {Key: "ConstantValues", Value: bson.A{int32(3), cv}}, {Key: "Name", Value: "Default"}} + part := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Settings$ConfigurationSettings"}, + {Key: "Configurations", Value: bson.A{int32(3), config}}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Settings$ProjectSettings"}, + {Key: "Settings", Value: bson.A{int32(2), part}}} +} + +// A configuration's constant override is a flat Value string below 10.9 and a +// Settings$SharedValue under SharedOrPrivateValue from it (mendixmodelsdk: +// Value deleted, SharedOrPrivateValue introduced 10.9.0). `mx convert` 9.24.30 +// turns the nested form into `{$ID, $Type, ConstantId, Value: ""}` — it does +// not know the nested value, so the override silently became empty, and mxcli +// refused it ("cannot store Settings$ConstantValue.SharedOrPrivateValue +// (Mendix 10.9.0+)"). +func TestNormalize_ConstantValueIsFlatBelowMendix10_9(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + flat bool + }{ + {pv(9, 24, 30), true}, + {pv(10, 8, 0), true}, + {pv(10, 9, 0), false}, + {pv(11, 13, 0), false}, // control + } { + for _, flat := range []string{"-", ""} { + d, err := normalizeShape(t, tt.v, settingsWithConstantValue("Settings$SharedValue", "kafka:9092", flat)) + if err != nil { + t.Errorf("%d.%d: a constant override is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + cv := find(d, "Settings$ConstantValue") + if !tt.flat { + if _, ok := get(cv, "SharedOrPrivateValue"); !ok { + t.Errorf("%d.%d: SharedOrPrivateValue was lowered: %v", tt.v.MajorVersion, tt.v.MinorVersion, keysOf(cv)) + } + continue + } + if got := strings.Join(keysOf(cv), ","); got != "$ID,$Type,ConstantId,Value" { + t.Errorf("%d.%d: constant value keys = %s, want $ID,$Type,ConstantId,Value", tt.v.MajorVersion, tt.v.MinorVersion, got) + } + if v, _ := get(cv, "Value"); v != "kafka:9092" { + t.Errorf("%d.%d: Value = %v, want kafka:9092", tt.v.MajorVersion, tt.v.MinorVersion, v) + } + } + } +} + +// A private value lives on the developer's workstation, which a flat Value +// cannot say: refused, naming the constant, not written as an empty override. +func TestNormalize_RefusesAPrivateConstantValueBelowMendix10_9(t *testing.T) { + _, err := normalizeShape(t, pv(9, 24, 30), settingsWithConstantValue("Settings$PrivateValue", "", "-")) + if err == nil || !strings.Contains(err.Error(), "M.ServerUrl") { + t.Errorf("a private constant value on 9.24 was not refused naming the constant: %v", err) + } +} + // Studio Pro's property set for a workflow on 9.24 includes the two state-change // event handlers Mendix removed in 11.0: `mx convert` 9.24.30 adds both as null // to a workflow mxcli wrote, as 10.24.24 does — a property-name difference From de548c61da5d2eee0d94864b18b162f016ff6240 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 17:58:53 +0000 Subject: [PATCH 08/44] fix(canon): leave a $currentObject flow argument out below Mendix 10.10 Symptom (ako/mxcli#1163, 29-datagrid DG12 on 9.24.30): a page passing `$currentObject` to a microflow or nanoflow was refused with "this Mendix 9.24.30 project cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". The same held on 10.0-10.9. Cause: bindParameterMappingValue writes such an argument as Expression text with Variable null. Expression is 10.10.0 on both mapping types. Before 10.10 a mapping holds only Variable, a Forms$PageVariable that is required there. Measured with the floor off: 9.24 does not load the page at all ("An error occurred when trying to set the 'Variable' property of a Microflow parameter mapping ... Value cannot be null"). Studio Pro 9 stores no mapping for an argument that is the enclosing object; it infers it. The 9.24.30 starter app and Workflow Commons 2.6.0 (9.24.0) have 82 flow calls taking 1-2 parameters and no mapping on any of them. Fix: below 10.10, canon.LowerToProjectShape leaves a $currentObject (or empty) argument out of every ParameterMappings list. An argument bound through Variable is kept; the floor drops its empty Expression. Any other expression has no form there, so it is refused, naming the parameter. Measured: bug-test 1163-flow-arguments-on-mendix9.mdl on a fresh 9.24.30 app: exec ok, mx check 0 errors, mx convert changes no key on the page, and describe -> exec is Unchanged. 29-datagrid on 9.24 now stops at DG15 (page variables, 10.17, a correct refusal); the DG2 NullReferenceException in mx check is the separate CE0463 gap. Controls: 29-datagrid PASS on 10.24.24 and 11.13.0. The unit tests fail on the unfixed code with the reported refusal. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-flow-argument-expression-below-10-10.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1163-flow-arguments-on-mendix9.mdl | 50 +++++++ modelsdk/canon/versionshape.go | 62 ++++++++- .../mpr/writer_version_shape_1163_test.go | 127 ++++++++++++++++++ 6 files changed, 241 insertions(+), 2 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-flow-argument-expression-below-10-10.json create mode 100644 mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-flow-argument-expression-below-10-10.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-flow-argument-expression-below-10-10.json new file mode 100644 index 000000000..78c6b027a --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-flow-argument-expression-below-10-10.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x and 10.0–10.9 a page passing `$currentObject` to a microflow or nanoflow was refused: `this Mendix 9.24.30 project cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)` (29-datagrid DG12 on 9.24)", "cause": "bindParameterMappingValue writes an argument that is not a page/snippet parameter or widget as Expression text with Variable null. Expression is 10.10.0; before it a mapping holds only Variable (a Forms$PageVariable, required before 10.10), and Studio Pro 9 does not store an argument that is the enclosing object at all — it infers it", "file": "`modelsdk/canon/versionshape.go` (LowerToProjectShape: lowerFlowArguments on any ParameterMappings list)", "insight": "**The floor's refusal hid a worse failure**: with the floor off, 9.24 does not load the project at all — `An error occurred when trying to set the 'Variable' property of a Microflow parameter mapping … ArgumentNullException` — so dropping only Expression would have shipped an unloadable page. **'Older shape' can be 'nothing stored'**: the 9.24 starter app and Workflow Commons 2.6.0 (9.24.0, from the Marketplace) have 82 flow calls taking 1–2 parameters and NOT ONE mapping (count calls against the called flow's parameters; an empty list alone proves nothing). So below 10.10 `$currentObject` (or an empty argument) is left out, a Variable-bound one keeps its Variable (the floor drops the empty Expression), and any other expression is refused naming the parameter. Readers needed nothing: describe already prints a call with no mappings as `call microflow M.F`, and re-executing that on 9.24 is Unchanged. Seen in passing: mxcli writes a populated Forms$CallNanoflowClientAction.ParameterMappings list with marker 3 (no RegisterListMarker for NanoflowParameterMapping), Studio Pro 2. Measured: bug-test on a fresh 9.24.30 app → exec ok, mx check 0 errors, mx convert changes no key; describe → exec Unchanged. 29-datagrid on 9.24 now stops at DG15 page variables (10.17, correct); 10.24.24 and 11.13 PASS. Tests: TestNormalize_FlowArgumentsBelowMendix10_10, TestNormalize_RefusesAnExpressionArgumentBelowMendix10_10; mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl", "refs": ["#1163", "#1140"], "ce": ["CE1571"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 5437a9653..2c79bb216 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **A page passing `$currentObject` to a microflow or nanoflow is written on Mendix 9 and on 10.0–10.9** (ako/mxcli#1163) — `call microflow M.F(P = $currentObject)` on a button or a data grid column was refused with "cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". An argument held as an expression arrived in 10.10. Before it, Studio Pro stores no argument for the enclosing object and infers it. Without the refusal, 9.24 could not even load the page, because the mapping had no `Variable`. Below 10.10 that argument is now left out, and a page parameter argument is stored as before. Any other expression argument is refused, naming the parameter. - **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. - **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). - **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 6256f085e..006e0596c 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -187,6 +187,7 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | | A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | +| A page's flow call argument that is the enclosing object (`call microflow M.F(P = $currentObject)`, `call nanoflow …`) | 10.10 | Not stored: Studio Pro infers it. A page parameter, snippet parameter or widget argument is stored as before, as a `Variable` | What an older shape cannot hold is refused, naming the element, rather than dropped: - a Spacing, colour-picker or toggle-button-group design property below 10.2; @@ -194,6 +195,7 @@ What an older shape cannot hold is refused, naming the element, rather than drop - a placeholder with parameters below 10.11; - a data view fed by another widget below 10.0; - a private constant value in a configuration below 10.9; +- a page's flow call argument that is any other expression (a literal, an attribute path) below 10.10; - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl b/mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl new file mode 100644 index 000000000..8934125f3 --- /dev/null +++ b/mdl-examples/bug-tests/1163-flow-arguments-on-mendix9.mdl @@ -0,0 +1,50 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x and on 10.0–10.9 a page passing +-- `$currentObject` to a microflow or nanoflow was refused: +-- +-- this Mendix 9.24.30 project cannot store +-- Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+) +-- +-- An argument stored as Expression text arrived in 10.10. Before it, a +-- mapping holds only a Variable (a Forms$PageVariable, required: a null one +-- makes 9.24 fail to load the page). An argument that is the enclosing object +-- is not stored at all, because Studio Pro 9.24 infers it: Workflow Commons +-- 2.6.0 (9.24.0) has 77 flow calls taking parameters and no mapping on any. +-- Below 10.10 mxcli therefore leaves `$currentObject` out and keeps a page +-- parameter as its Variable. Any other expression (a literal, an attribute +-- path) has no form there and is refused, naming the parameter. +-- +-- Run it on a fresh 9.24.30 app (`mxcli new … --version 9.24.30.51827`): exec, +-- then `mxcli docker check` reports the blank app's error count, and `mx +-- convert -p` changes no key on the page. + +create module Probe1163Args; + +create persistent entity Probe1163Args.Item ( + Name: String(100) +); + +create microflow Probe1163Args.ACT_Item ( $Item: Probe1163Args.Item ) +begin + log info node 'Probe1163Args' 'item'; +end; + +create nanoflow Probe1163Args.NF_Item ( $Item: Probe1163Args.Item ) +begin + log info node 'Probe1163Args' 'item'; +end; + +create page Probe1163Args.Item_Edit ( + Title: 'Item', + Layout: Atlas_Core.Atlas_Default, + Params: ( $Item: Probe1163Args.Item ) +) { + dataview dv (DataSource: $Item) { + actionbutton btnCurrent (Caption: 'Current', + Action: call microflow Probe1163Args.ACT_Item(Item = $currentObject)) + actionbutton btnNanoflow (Caption: 'Nanoflow', + Action: call nanoflow Probe1163Args.NF_Item(Item = $currentObject)) + } + actionbutton btnParam (Caption: 'Param', + Action: call microflow Probe1163Args.ACT_Item(Item = $Item)) +}; diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 399afe049..0ee201248 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -42,6 +42,13 @@ import ( // from 10.9.0; before, the value as a flat Value string, `{$ID, $Type, // ConstantId, Value}` (`mx convert` 9.24.30, which empties the nested form // it does not know). +// - Forms$MicroflowParameterMapping / NanoflowParameterMapping: an argument +// as Expression text from 10.10.0; before, only a Variable (a required +// Forms$PageVariable), and an argument that is the enclosing object is not +// stored at all — Studio Pro 9.24 infers it (Workflow Commons 2.6.0: 77 +// flow calls taking parameters, no mapping on any). mxcli writes that +// argument as Expression `$currentObject` with Variable null, which 9.24 +// cannot even load. // - Workflows$ PersistentId, from 10.21.0 on the workflow and on the abstract // WorkflowActivity, Outcome and BoundaryEvent (ako/mxcli#1163). Workflows // exist from 9.0; `mx convert` 9.24.30 strips every one. The workflow's @@ -64,9 +71,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { noFullObjects: !PropertyDeclared("Forms$DataSource", "ForceFullObjects", *pv), noPersistentID: !PropertyDeclared("Workflows$Workflow", "PersistentId", *pv), flatConstant: !PropertyDeclared("Settings$ConstantValue", "SharedOrPrivateValue", *pv), + variableArgs: !PropertyDeclared("Forms$MicroflowParameterMapping", "Expression", *pv), } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID && !l.flatConstant { + !l.noPersistentID && !l.flatConstant && !l.variableArgs { return contents, nil } var doc bson.D @@ -87,7 +95,7 @@ type shapeLowering struct { pv version.Version flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool - flatConstant bool + flatConstant, variableArgs bool changed bool } @@ -132,6 +140,11 @@ func (l *shapeLowering) walk(v any) (any, error) { if l.noPersistentID && strings.HasPrefix(typ, "Workflows$") { x = l.dropPersistentID(x) } + if l.variableArgs { + if err = l.lowerFlowArguments(x); err != nil { + return nil, err + } + } for i := range x { if x[i].Value, err = l.walk(x[i].Value); err != nil { return nil, err @@ -307,6 +320,51 @@ func (l *shapeLowering) lowerConstantValue(d bson.D) (bson.D, error) { return d, nil } +var flowParameterMappings = map[string]bool{ + "Forms$MicroflowParameterMapping": true, "Forms$NanoflowParameterMapping": true, +} + +// lowerFlowArguments rewrites the flow call arguments in d's ParameterMappings +// list, if it has one, into what a version before 10.10 stores: an argument +// bound through Variable keeps it (its empty Expression is the floor's to +// drop); an argument that is the enclosing object (`$currentObject`) — or one +// that binds nothing — is left out, which is how Studio Pro 9.24 stores it. +// Any other expression has no Variable form and is refused, naming the +// parameter. +func (l *shapeLowering) lowerFlowArguments(d bson.D) error { + i := shapeIndex(d, "ParameterMappings") + if i < 0 { + return nil + } + list, _ := d[i].Value.(bson.A) + out := bson.A{} + dropped := false + for _, el := range list { + m, ok := el.(bson.D) + typ, _ := shapeString(m, "$Type") + if !ok || !flowParameterMappings[typ] { + out = append(out, el) + continue + } + if v, _ := shapeValue(m, "Variable"); v != nil { + out = append(out, el) + continue + } + expr, _ := shapeString(m, "Expression") + if e := strings.TrimSpace(expr); e != "" && !strings.EqualFold(e, "$currentObject") { + param, _ := shapeString(m, "Parameter") + return fmt.Errorf("the argument for %s is the expression %s, which Mendix introduced in 10.10 (this project is %s): "+ + "an argument there is the enclosing object, or a page parameter, snippet parameter or widget", param, expr, l.pv) + } + dropped = true + } + if dropped { + d[i].Value = out + l.changed = true + } + return nil +} + // dropPersistentID removes a workflow element's PersistentId below 10.21, where // none declares it. Whatever it holds is dropped, not refused: a PersistentId is // minted by the writer (a fresh GUID on every build), never stated by a script, diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index eb00c2189..3d33cfcb7 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -235,6 +235,133 @@ func TestNormalize_RefusesAPrivateConstantValueBelowMendix10_9(t *testing.T) { } } +// flowMapping is one argument of a page's flow call as mxcli builds it +// (bindParameterMappingValue): a page-variable reference under Variable with an +// empty Expression, anything else as Expression text with Variable null. +func flowMapping(mappingType, param, expression, pageParam string) bson.D { + var variable any + if pageParam != "" { + variable = bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$PageVariable"}, + {Key: "PageParameter", Value: pageParam}, {Key: "SnippetParameter", Value: ""}, + {Key: "UseAllPages", Value: false}, {Key: "Widget", Value: ""}} + } + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: mappingType}, + {Key: "Expression", Value: expression}, {Key: "Parameter", Value: param}, {Key: "Variable", Value: variable}} +} + +// pageWithFlowCalls is a page with a microflow button and a nanoflow button, +// each passing the given mappings. +func pageWithFlowCalls(microflowMappings, nanoflowMappings []bson.D) bson.D { + list := func(ms []bson.D) bson.A { + a := bson.A{int32(2)} + for _, m := range ms { + a = append(a, m) + } + return a + } + mfButton := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$ActionButton"}, {Key: "Name", Value: "btnMf"}, + {Key: "Action", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$MicroflowAction"}, + {Key: "MicroflowSettings", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$MicroflowSettings"}, + {Key: "Microflow", Value: "M.ACT"}, {Key: "ParameterMappings", Value: list(microflowMappings)}}}}}} + nfButton := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$ActionButton"}, {Key: "Name", Value: "btnNf"}, + {Key: "Action", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$CallNanoflowClientAction"}, + {Key: "Nanoflow", Value: "M.NF"}, {Key: "ParameterMappings", Value: list(nanoflowMappings)}}}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$Page"}, {Key: "Name", Value: "P"}, + {Key: "Widgets", Value: bson.A{int32(2), mfButton, nfButton}}} +} + +// mappingsOf returns the mappings stored under the first $Type holder in d. +func mappingsOf(t *testing.T, d bson.D, holder string) []bson.D { + t.Helper() + list, _ := get(find(d, holder), "ParameterMappings") + var out []bson.D + for _, el := range listItems(list) { + out = append(out, el.(bson.D)) + } + return out +} + +func listItems(v any) bson.A { + a, _ := v.(bson.A) + if len(a) > 0 { + if _, marker := a[0].(int32); marker { + return a[1:] + } + } + return a +} + +// A flow argument on a page is a Variable (a Forms$PageVariable, required) below +// 10.10, and Expression text only from 10.10 (mendixmodelsdk). mxcli writes +// `$currentObject` as Expression with Variable null: the floor refused it on +// 9.24 ("cannot store Forms$MicroflowParameterMapping.Expression (Mendix +// 10.10.0+)"), and without the floor 9.24 does not load the page at all ("An +// error occurred when trying to set the 'Variable' property of a Microflow +// parameter mapping … Value cannot be null"). Studio Pro 9.24 stores no mapping +// for an argument that is the enclosing object — it infers it: Workflow +// Commons 2.6.0 (9.24.0), 75 microflow and 2 nanoflow calls taking parameters, +// none with a mapping; the 9.24.30 starter app's ShowPasswordForm(Account) +// button likewise. So below 10.10 that mapping is not written; a page +// parameter keeps its Variable and loses only the empty Expression. +func TestNormalize_FlowArgumentsBelowMendix10_10(t *testing.T) { + build := func() bson.D { + return pageWithFlowCalls( + []bson.D{flowMapping("Forms$MicroflowParameterMapping", "M.ACT.Item", "$currentObject", ""), + flowMapping("Forms$MicroflowParameterMapping", "M.ACT.Order", "", "Order")}, + []bson.D{flowMapping("Forms$NanoflowParameterMapping", "M.NF.Item", "$currentObject", "")}) + } + for _, tt := range []struct { + v *version.ProjectVersion + lowered bool + mfCount int + nfCount int + mfParams string + }{ + {pv(9, 24, 30), true, 1, 0, "M.ACT.Order"}, + {pv(10, 9, 0), true, 1, 0, "M.ACT.Order"}, + {pv(10, 10, 0), false, 2, 1, "M.ACT.Item,M.ACT.Order"}, + {pv(11, 13, 0), false, 2, 1, "M.ACT.Item,M.ACT.Order"}, // control + } { + d, err := normalizeShape(t, tt.v, build()) + if err != nil { + t.Errorf("%d.%d: the page is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + mf := mappingsOf(t, d, "Forms$MicroflowSettings") + nf := mappingsOf(t, d, "Forms$CallNanoflowClientAction") + var params []string + for _, m := range mf { + p, _ := get(m, "Parameter") + params = append(params, p.(string)) + if _, ok := get(m, "Expression"); ok == tt.lowered { + t.Errorf("%d.%d: mapping %v keeps Expression %v", tt.v.MajorVersion, tt.v.MinorVersion, p, ok) + } + if v, _ := get(m, "Variable"); tt.lowered && v == nil { + t.Errorf("%d.%d: mapping %v has no Variable, which 9.x cannot load", tt.v.MajorVersion, tt.v.MinorVersion, p) + } + } + if len(mf) != tt.mfCount || len(nf) != tt.nfCount || strings.Join(params, ",") != tt.mfParams { + t.Errorf("%d.%d: %d microflow mappings (%s) and %d nanoflow mappings, want %d (%s) and %d", + tt.v.MajorVersion, tt.v.MinorVersion, len(mf), strings.Join(params, ","), len(nf), tt.mfCount, tt.mfParams, tt.nfCount) + } + } +} + +// Any other expression — a literal, an attribute path — has no Variable form: +// refused below 10.10, naming the parameter, rather than dropped. +func TestNormalize_RefusesAnExpressionArgumentBelowMendix10_10(t *testing.T) { + for _, m := range []bson.D{ + flowMapping("Forms$MicroflowParameterMapping", "M.ACT.Flag", "true", ""), + flowMapping("Forms$NanoflowParameterMapping", "M.NF.Name", "$Order/Name", ""), + } { + _, err := normalizeShape(t, pv(9, 24, 30), pageWithFlowCalls([]bson.D{m}, nil)) + p, _ := get(m, "Parameter") + if err == nil || !strings.Contains(err.Error(), p.(string)) || !strings.Contains(err.Error(), "10.10") { + t.Errorf("an expression argument for %v on 9.24 was not refused naming it: %v", p, err) + } + } +} + // Studio Pro's property set for a workflow on 9.24 includes the two state-change // event handlers Mendix removed in 11.0: `mx convert` 9.24.30 adds both as null // to a workflow mxcli wrote, as 10.24.24 does — a property-name difference From baaeda16f358a5497b953f317c96eaaf7b78010d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 18:07:21 +0000 Subject: [PATCH 09/44] fix(executor): gate create published odata service at Mendix 10.4 Symptom (ako/mxcli#1163, 10-odata and 18-folder on 9.24.30): a published OData service was refused at write time with "this Mendix 9.24.30 project cannot store ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+)". By then the script's earlier statements were already written. Cause: not the property. mxcli writes the service as ODataPublish$PublishedODataService2, a document type mendixmodelsdk introduces in 10.4.0. Before 10.4 a published OData service is a Rest$PublishedODataService, which is deleted in 10.4.0: a different document with its own children, and mxcli does not write it. The generated version data has property floors only, so the floor tripped over the first versioned key. Measured with the floor off: mx check 9.24.30 cannot load the project ("TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name ODataPublish$PublishedODataService2"). Lowering the property would have made an unloadable app. Fix: a version gate. `create [or modify] published odata service` needs integration.published_odata_service, 10.4.0. `check -p` and exec's pre-flight report it as MDL-VERSION01 before anything is written, with a hint naming the 9.x document. Writing Rest$PublishedODataService for 9.x is not attempted here. Measured: bug-test 1163-published-odata-below-10-4.mdl with `check -p` on a fresh 9.24.30 app reports MDL-VERSION01. 10-odata and 18-folder on 9.24 now stop at the gate. Controls: both PASS on 10.24.24 and 11.13.0. The new gate cases fail without the gate (0 violations). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...28-published-odata-is-a-10-4-document.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1163-published-odata-below-10-4.mdl | 42 +++++++++++++++++++ mdl/executor/version_gates.go | 11 ++++- mdl/executor/version_gates_test.go | 11 ++++- sdk/versions/mendix-10.yaml | 11 +++++ sdk/versions/mendix-11.yaml | 4 ++ 8 files changed, 79 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-28-published-odata-is-a-10-4-document.json create mode 100644 mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-published-odata-is-a-10-4-document.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-published-odata-is-a-10-4-document.json new file mode 100644 index 000000000..87ed75f19 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-published-odata-is-a-10-4-document.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-28", "symptom": "On Mendix 9.x `create published odata service` was refused at write time with `this Mendix 9.24.30 project cannot store ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+)` (10-odata and 18-folder on 9.24), after the earlier statements of the script were already written", "cause": "mxcli writes a published OData service as ODataPublish$PublishedODataService2, a document type introduced in 10.4.0; before it the service is Rest$PublishedODataService (deleted 10.4.0), a different document with its own children. The generated version data has no type-level floor, so the property floor could only trip over the first versioned key and named the wrong thing", "file": "`mdl/executor/version_gates.go` (CreateODataServiceStmt gate), `sdk/versions/mendix-10.yaml` / `mendix-11.yaml` (integration.published_odata_service 10.4.0)", "insight": "**Before lowering a refused property, check the type it sits on exists in that version.** Lowering IncludeMetadataByDefault was the obvious fix and would have made it worse: with the floor off, mx check 9.24.30 stops loading the project at all (`TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name ODataPublish$PublishedODataService2`). mendixmodelsdk says it directly: `PublishedODataService2.versionInfo = {introduced: \"10.4.0\"}`, `rest.PublishedODataService` `deleted: \"10.4.0\"`. The generated tables carry property floors only, so a whole-type floor has to be a version gate (check-time, MDL-VERSION01, before anything is written) — the same gap lets Workflows$EndOfParallelSplitPathActivity (10.21) through on 9.x. Measured: 10-odata and 18-folder on 9.24 now fail at the gate (no write); on 10.24.24 and 11.13 both PASS. Tests: TestCheckVersionGates_ReportsEveryHandlerGate (published odata on 9.24.30 and 10.3), TestCheckVersionGates_UngatedFormsPass (10.4 passes); mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl", "refs": ["#1163", "#1434"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c79bb216..8982c988e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. +- **`create published odata service` on Mendix 9 and 10.0–10.3 is refused up front, naming the real reason** (ako/mxcli#1163) — it was refused at write time for "ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+)". That property was not the problem: the whole document mxcli writes is a 10.4 type, and Mendix 9.24 cannot load a project that holds one. Before 10.4 a published OData service is a different document that mxcli does not write. `check -p` and `exec` now report MDL-VERSION01 before writing anything. - **A page passing `$currentObject` to a microflow or nanoflow is written on Mendix 9 and on 10.0–10.9** (ako/mxcli#1163) — `call microflow M.F(P = $currentObject)` on a button or a data grid column was refused with "cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". An argument held as an expression arrived in 10.10. Before it, Studio Pro stores no argument for the enclosing object and infers it. Without the refusal, 9.24 could not even load the page, because the mapping had no `Variable`. Below 10.10 that argument is now left out, and a page parameter argument is stored as before. Any other expression argument is refused, naming the parameter. - **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. - **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 006e0596c..e6b4cd22a 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -200,6 +200,8 @@ What an older shape cannot hold is refused, naming the element, rather than drop Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. +A published OData service is not an older shape of the same document. Before 10.4 it is a different document (`Rest$PublishedODataService`, not `ODataPublish$PublishedODataService2`), which mxcli does not write. `create published odata service` is refused below 10.4 with MDL-VERSION01, by `check -p` and by `exec` before anything is written. Mendix 9.24 cannot load a project that holds the 10.4 document. + ### Widget Type Definitions Pluggable widget PropertyTypes change between Mendix versions. A widget template extracted from 11.6 may have more or fewer properties than the widget installed in a 10.24 project. The [widget augmentation system](../internals/widget-templates.md) handles this by reconciling templates against the project's `.mpk` files. diff --git a/mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl b/mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl new file mode 100644 index 000000000..d42e11fea --- /dev/null +++ b/mdl-examples/bug-tests/1163-published-odata-below-10-4.mdl @@ -0,0 +1,42 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x a published OData service was refused with +-- a message naming the wrong thing: +-- +-- this Mendix 9.24.30 project cannot store +-- ODataPublish$PublishedODataService2.IncludeMetadataByDefault (Mendix 10.8.0+) +-- +-- That property is not the problem. The whole document mxcli writes, +-- ODataPublish$PublishedODataService2, was introduced in 10.4.0 +-- (mendixmodelsdk). Before 10.4 a published OData service is a +-- Rest$PublishedODataService, a different document that mxcli does not +-- write. With the floor off, mx check 9.24.30 cannot load the project +-- ("TypeCacheUnknownTypeException: ... ODataPublish$PublishedODataService2"), +-- so lowering the property would have produced an unloadable app. +-- +-- `mxcli check -p <9.24 app>` now reports MDL-VERSION01 "create +-- published odata service" before anything is written, and exec refuses the +-- same way. On 10.4 and later the service is written as before. + +create module Probe1163OData; + +create persistent entity Probe1163OData.Customer ( + Name: String(100), + Email: String(200) unique +); + +create published odata service Probe1163OData.CustomerAPI ( + Path: 'odata/customers/', + Version: '1.0.0', + ODataVersion: OData4, + Namespace: 'Probe1163OData.Customers', + ServiceName: 'Customer Service' +) +{ + publish entity Probe1163OData.Customer as 'Customers' ( + ReadMode: source + ) + expose ( + Email as 'customerId' (KEY, Filterable, Sortable), + Name + ); +}; diff --git a/mdl/executor/version_gates.go b/mdl/executor/version_gates.go index 8d6fe0ae5..33514acad 100644 --- a/mdl/executor/version_gates.go +++ b/mdl/executor/version_gates.go @@ -53,11 +53,18 @@ func statementVersionGates(stmt ast.Statement) []versionGate { return []versionGate{{"integration", "published_rest_grant_revoke", "grant access on published rest service", "upgrade your project to 10.0+"}} case *ast.CreateODataServiceStmt: + // The document mxcli writes, ODataPublish$PublishedODataService2, is + // 10.4+; before, a published OData service is a Rest$PublishedODataService, + // a different document mxcli does not write (ako/mxcli#1163). + gates := []versionGate{{"integration", "published_odata_service", "create published odata service", + "Before Mendix 10.4 a published OData service is a different document " + + "(Rest$PublishedODataService), which mxcli does not write. Publish it in Studio Pro, or upgrade the project."}} if s.SupportsGraphQL { - return []versionGate{{"integration", "odata_graphql", "SupportsGraphQL on a published OData service", + gates = append(gates, versionGate{"integration", "odata_graphql", "SupportsGraphQL on a published OData service", "Publishing a service as GraphQL as well arrived in Studio Pro 10.14. " + - "Remove SupportsGraphQL to publish OData only."}} + "Remove SupportsGraphQL to publish OData only."}) } + return gates case *ast.CreateMessageDefinitionStmt: return []versionGate{messageDefinitionDocumentGate("create message definition")} case *ast.AlterSettingsStmt: diff --git a/mdl/executor/version_gates_test.go b/mdl/executor/version_gates_test.go index 18b555be7..7b320b2e0 100644 --- a/mdl/executor/version_gates_test.go +++ b/mdl/executor/version_gates_test.go @@ -32,6 +32,10 @@ func TestCheckVersionGates_ReportsEveryHandlerGate(t *testing.T) { {"alter published rest", &ast.AlterPublishedRestServiceStmt{}, [3]int{9, 24, 0}, "alter published rest service"}, {"grant published rest", &ast.GrantPublishedRestServiceAccessStmt{}, [3]int{9, 24, 0}, "grant access on published rest service"}, {"odata graphql", &ast.CreateODataServiceStmt{SupportsGraphQL: true}, [3]int{10, 13, 0}, "SupportsGraphQL"}, + // ako/mxcli#1163: the document mxcli writes is 10.4+; 9.24.30 does not + // load a project holding one (TypeCacheUnknownTypeException). + {"published odata", &ast.CreateODataServiceStmt{}, [3]int{9, 24, 30}, "create published odata service"}, + {"published odata 10.3", &ast.CreateODataServiceStmt{}, [3]int{10, 3, 0}, "create published odata service"}, {"message definition document", &ast.CreateMessageDefinitionStmt{}, [3]int{11, 14, 0}, "create message definition"}, {"workflow group", &ast.AlterSettingsStmt{Section: "WORKFLOWS", AddGroup: true, GroupName: "G"}, [3]int{11, 1, 0}, "add group"}, {"view entity", &ast.CreateViewEntityStmt{}, [3]int{10, 17, 0}, "create view entity"}, @@ -104,14 +108,13 @@ func TestCheckVersionGates_ReportsEveryHandlerGate(t *testing.T) { // Statements with no gate, or a gated statement without the gated clause, are // not reported — `synchronize all` on 9.0, a page without parameters, OData -// without GraphQL. +// without GraphQL from 10.4. func TestCheckVersionGates_UngatedFormsPass(t *testing.T) { prog := &ast.Program{Statements: []ast.Statement{ &ast.CreateModuleStmt{Name: "M"}, &ast.CreateMicroflowStmt{}, &ast.CreateNanoflowStmt{Body: []ast.MicroflowStatement{&ast.SynchronizeStmt{SyncType: "All"}}}, &ast.CreatePageStmtV3{}, - &ast.CreateODataServiceStmt{}, &ast.AlterSettingsStmt{Section: "MODEL"}, // No EXPORT LEVEL clause: the stored level is carried (#1498). &ast.CreateEntityStmt{Attributes: []ast.Attribute{{Name: "A"}}}, @@ -122,6 +125,10 @@ func TestCheckVersionGates_UngatedFormsPass(t *testing.T) { if vs := versionGateViolations(versionCtx(t, 9, 0, 0), prog); len(vs) != 0 { t.Errorf("ungated forms reported on 9.0: %+v", vs) } + odata := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, &ast.CreateODataServiceStmt{}}} + if vs := versionGateViolations(versionCtx(t, 10, 4, 0), odata); len(vs) != 0 { + t.Errorf("a published OData service without GraphQL reported on 10.4: %+v", vs) + } } // One source of truth: a handler that called checkFeature with its own feature diff --git a/sdk/versions/mendix-10.yaml b/sdk/versions/mendix-10.yaml index adebf0d35..207443cab 100644 --- a/sdk/versions/mendix-10.yaml +++ b/sdk/versions/mendix-10.yaml @@ -167,6 +167,17 @@ features: mdl: "CREATE BUSINESS EVENT SERVICE Module.Name ..." odata_client: min_version: "10.0.0" + published_odata_service: + # mxcli writes a published OData service as ODataPublish$PublishedODataService2, + # introduced in 10.4.0 (mendixmodelsdk); before, it is a + # Rest$PublishedODataService, a different document with its own children, + # which mxcli does not write. Measured: mx check 9.24.30 cannot load a + # project holding a PublishedODataService2 — + # "TypeCacheUnknownTypeException: ... ODataPublish$PublishedODataService2" + # (ako/mxcli#1163). Below this the property floor used to refuse + # IncludeMetadataByDefault instead, which named the wrong thing. + min_version: "10.4.0" + mdl: "CREATE PUBLISHED ODATA SERVICE Module.Name (...) { PUBLISH ENTITY ... }" odata_graphql: # A published OData service can also answer GraphQL — one boolean, same # resources, an extra endpoint. Introduced in Studio Pro 10.14 as an diff --git a/sdk/versions/mendix-11.yaml b/sdk/versions/mendix-11.yaml index 687a18b3b..02a08812e 100644 --- a/sdk/versions/mendix-11.yaml +++ b/sdk/versions/mendix-11.yaml @@ -179,6 +179,10 @@ features: mdl: "CREATE BUSINESS EVENT SERVICE Module.Name ..." odata_client: min_version: "10.0.0" + published_odata_service: + # ODataPublish$PublishedODataService2 is 10.4+; see mendix-10.yaml. + min_version: "10.4.0" + mdl: "CREATE PUBLISHED ODATA SERVICE Module.Name (...) { PUBLISH ENTITY ... }" odata_graphql: # A published OData service can also answer GraphQL — one boolean, same # resources, an extra endpoint. Introduced in Studio Pro 10.14 as an From c190ea705e6c231cd27ae1316d18320942e35262 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 18:07:44 +0000 Subject: [PATCH 10/44] docs(wiki): a version refusal is not always an older shape The 9.24 doctype run for ako/mxcli#1163 split five refusals three ways: a wrong floor in the generated data (AggregateFunction), a document type the version lacks (PublishedODataService2, where lowering would have produced an unloadable project), and an older shape that stores nothing (a flow argument that is the enclosing object). Record what settled each one: the SDK's version data, Studio Pro-authored bytes from a Marketplace .mpk, and `mx convert -p` on the older version against a floor-off build, comparing values as well as keys. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .../bug-patterns/the-platform-ci-does-not-run.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs-wiki/bug-patterns/the-platform-ci-does-not-run.md b/docs-wiki/bug-patterns/the-platform-ci-does-not-run.md index 1b1116a8d..8e4c471e1 100644 --- a/docs-wiki/bug-patterns/the-platform-ci-does-not-run.md +++ b/docs-wiki/bug-patterns/the-platform-ci-does-not-run.md @@ -91,6 +91,18 @@ every unit write passes (`canon.LowerToProjectShape`, before the floor), and to read both shapes. A starter app is the cheapest source of the older shape's real bytes: `mxcli new --version 9.24.30.51827` authored by Studio Pro. +A refusal is not always an older shape, though. In the next 9.24 run (#1163), one +of five refusals was the floor's own data being wrong: an aggregate's function had +been given the row of the fold `reduce` added in 10.0. One was a document type the +version does not have: a published OData service before 10.4. That needs a gate, +because lowering its property would have produced a project that does not load. +For one more, the older shape was "nothing stored": Studio Pro 9 infers a flow +argument that is the enclosing object. Three things settled each one: the SDK's +version data, Studio Pro-authored bytes (a Marketplace `.mpk` holds a whole +`project.mpr` of its Mendix version), and `mx convert -p` on the older version +run against a floor-off build. Convert strips undeclared keys and empties values +it does not understand, so compare values as well as key sets. + **mxcli sometimes generates the environment it then fails in.** The non-root devcontainer that could not provision PostgreSQL was scaffolded by `mxcli new` — the same code emits that base image, installs postgresql, and runs as `vscode`. From 42778ae484e3125a7e8074808f08ffad2b1dc372 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 19:49:15 +0000 Subject: [PATCH 11/44] fix(test): git doctor tests no longer read the developer's git config `git remote get-url` applies url..insteadOf, and doctorEnv left the system, global and GIT_CONFIG_COUNT/KEY/VALUE configuration in place. In a sandbox that rewrites git@github.com: to https://github.com/ through the environment, the SSH remote the tests add already read back as HTTPS, and TestGitDoctor_SSHRemoteRewrittenByFix and TestGitDoctor_SSHURLFormRewrittenByFix failed `make test` on main (#1670) while CI was green. A ~/.gitconfig rewriting https to git@ fails the same two tests the other way. doctorEnv now sets GIT_CONFIG_GLOBAL to the null device, GIT_CONFIG_NOSYSTEM=1 and GIT_CONFIG_COUNT=0. Control: before the fix both tests fail under each rewrite separately (environment, and HOME with an https->git@ insteadOf); with an empty HOME and no environment rewrite they pass. After it, both rewrites pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...026-10-28-git-doctor-tests-read-ambient-insteadof.json | 1 + cmd/mxcli/gitdoctor_test.go | 8 ++++++++ 2 files changed, 9 insertions(+) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-tests-read-ambient-insteadof.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-tests-read-ambient-insteadof.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-tests-read-ambient-insteadof.json new file mode 100644 index 000000000..85b1854a1 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-tests-read-ambient-insteadof.json @@ -0,0 +1 @@ +{"area":"cmd/mxcli","date":"2026-10-28","symptom":"`make test` failed on main right after #1670 (git doctor) merged, in a hosted sandbox only: TestGitDoctor_SSHRemoteRewrittenByFix and TestGitDoctor_SSHURLFormRewrittenByFix reported `remote url = \"https://github.com/org/app.git\\n\", want https://github.com/org/app`; CI was green","cause":"`git remote get-url` applies url..insteadOf, and the tests ran git under the developer's own configuration. The sandbox sets GIT_CONFIG_COUNT/KEY/VALUE rewriting git@github.com: and ssh://git@github.com/ to https://github.com/, so the SSH remote the test added already read back as HTTPS and --fix never touched it; a ~/.gitconfig rewriting https://github.com/ to git@github.com: (the usual developer setup) fails the same two tests the other way, `git@github.com:org/app`","file":"`cmd/mxcli/gitdoctor_test.go` (`doctorEnv`: GIT_CONFIG_GLOBAL=os.DevNull, GIT_CONFIG_NOSYSTEM=1, GIT_CONFIG_COUNT=0)","insight":"A test that shells out to git inherits three configuration sources besides the repository's — system, global, and the GIT_CONFIG_COUNT environment — and isolating HOME or the identity variables covers none of the third. Reproduce each source separately, because they fail in opposite directions: env rewrite on this host, then `env -u GIT_CONFIG_* HOME=git@ insteadOf>`, with an empty HOME as the passing control. Both failed before the fix and pass after. Not fixed, and a product question rather than a test one: with a global https->git@ rewrite, `git doctor` reads the effective URL, so it reports SSH and `--fix` sets an HTTPS URL that the rewrite turns straight back into SSH, every run","refs":["ako/mxcli#1670"]} diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index c0e0393f4..fb2cf56c1 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -23,6 +23,14 @@ import ( func doctorEnv(t *testing.T, dir string) { t.Helper() requireGit(t) + // The repository under test is the only git configuration these tests may + // see. `git remote get-url` applies url..insteadOf, so a rewrite in + // ~/.gitconfig (https -> git@, common on developer machines) or in the + // GIT_CONFIG_COUNT/KEY/VALUE environment (git@ -> https, set by hosted + // sandboxes) changed the URL the remote tests read back. + t.Setenv("GIT_CONFIG_GLOBAL", os.DevNull) + t.Setenv("GIT_CONFIG_NOSYSTEM", "1") + t.Setenv("GIT_CONFIG_COUNT", "0") t.Setenv("GIT_CEILING_DIRECTORIES", filepath.Dir(dir)) t.Setenv("GIT_AUTHOR_NAME", "t") t.Setenv("GIT_AUTHOR_EMAIL", "t@example.com") From a6f3dfad1255de0a084fa7ce91d60e19c42d891d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 19:29:12 +0000 Subject: [PATCH 12/44] fix(make): fmt-check uses the toolchain's gofmt, not the one on PATH go.mod's toolchain line makes `go` (and `make fmt`) switch to go1.27.3, but fmt-check ran a bare `gofmt`, which is whatever Go is installed. With go1.24.7 installed, `make lint` failed main on a file CI accepts (mdl/backend/mfmutator/splice_handler_test.go) and passed a file CI then rejected (ako/mxcli#1759). GOFMT defaults to $(go env GOROOT)/bin/gofmt. Control: `make fmt-check GOFMT=/usr/local/go/bin/gofmt` still fails on splice_handler_test.go; the default passes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- Makefile | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 12f80861e..4af1eae41 100644 --- a/Makefile +++ b/Makefile @@ -470,13 +470,21 @@ lint-go: fmt-check vet # Tracked files only: the ANTLR parser under mdl/grammar/parser is generated at # build time and deliberately not committed. The empty-list guard is a positive # control -- a check that inspected nothing must fail loudly rather than pass. +# +# The gofmt is the toolchain's, not the first one on PATH. go.mod's `toolchain` +# line makes `go` (and so `make fmt`) switch to that release, but a bare `gofmt` +# is whatever Go is installed, and gofmt's output changes between releases. With +# Go 1.24.7 installed and the toolchain at 1.27.3, the bare one failed main on a +# file CI accepts (mdl/backend/mfmutator/splice_handler_test.go) and passed a +# file CI then rejected (ako/mxcli#1759). +GOFMT ?= $(shell go env GOROOT)/bin/gofmt fmt-check: @files=$$(git ls-files '*.go' | grep -v '/parser/'); \ if [ -z "$$files" ]; then \ echo "fmt-check: found no Go files to check -- the file list is wrong"; \ exit 1; \ fi; \ - unformatted=$$(gofmt -l $$files); \ + unformatted=$$($(GOFMT) -l $$files); \ if [ -n "$$unformatted" ]; then \ echo "Not gofmt-clean (run 'make fmt'):"; \ echo "$$unformatted"; \ From f4398e6652509db7b981d7e9881b7eed818e2e63 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 19:29:12 +0000 Subject: [PATCH 13/44] build(make): add ci-local, the build-and-test job run before pushing `make build test lint` covers less than CI's build-and-test job gates on: check-mdl, check-findings, check-skill-index, check-wiki-pages and the rest ran only in CI unless someone remembered them. ci-local runs the job's steps in its order (check-skill-mdl standing for its three skill/docs MDL steps), without govulncheck, which the job installs @latest over the network, and without the integration jobs, which need mxbuild. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- Makefile | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 4af1eae41..f64a174fa 100644 --- a/Makefile +++ b/Makefile @@ -8,6 +8,7 @@ # make test-integration - Run integration tests (requires mx/mxbuild) # make test-mdl - Run MDL integration tests (requires Docker) # make lint - Lint all code (Go + TypeScript) +# make ci-local - Run what CI's build-and-test job runs, before pushing # make lint-go - Lint Go code (fmt + vet) # make lint-ts - Lint TypeScript code (tsc --noEmit) # make grammar - Regenerate ANTLR parser @@ -35,7 +36,7 @@ GO_BUILD_FLAGS = -trimpath # Clean version for VS Code extension (must be valid semver: major.minor.patch) VSCE_VERSION = $(shell echo "$(VERSION)" | sed 's/^v//; s/-.*//' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$$' || echo "0.0.0") -.PHONY: build build-debug size release clean test test-mdl check-mdl check-skill-mdl check-conformance conformance-shrink gen-migration-reference check-migration-reference check-skill-pack-js check-findings check-wiki-pages digest-status check-tunnel-deps check-test-timeouts check-widget-versions test-integration test-integration-executor test-integration-roundtrip test-integration-parity test-integration-upgrade test-integration-other grammar completions sync-skills sync-skill-packs sync-commands sync-mods sync-lint-rules sync-changelog sync-all docs documentation docs-site docs-serve vscode-ext vscode-install source-tree sbom sbom-report lint lint-go lint-ts fmt fmt-check vet skill-index check-skill-index +.PHONY: build build-debug size release clean test ci-local test-mdl check-mdl check-skill-mdl check-conformance conformance-shrink gen-migration-reference check-migration-reference check-skill-pack-js check-findings check-wiki-pages digest-status check-tunnel-deps check-test-timeouts check-widget-versions test-integration test-integration-executor test-integration-roundtrip test-integration-parity test-integration-upgrade test-integration-other grammar completions sync-skills sync-skill-packs sync-commands sync-mods sync-lint-rules sync-changelog sync-all docs documentation docs-site docs-serve vscode-ext vscode-install source-tree sbom sbom-report lint lint-go lint-ts fmt fmt-check vet skill-index check-skill-index # Helper: copy file only if content differs (avoids mtime updates that invalidate go build cache) # Usage: $(call copy-if-changed,src,dst) @@ -453,6 +454,20 @@ check-widget-versions: build # Lint all code (Go + TypeScript) lint: lint-go lint-ts check-conformance check-test-timeouts +# What the build-and-test job in .github/workflows/push-test.yml runs, in its +# order, so a push is proven against the gate it will meet rather than against +# `build test lint`, which covers less of it: check-mdl, check-findings, +# check-skill-index and the rest run only in CI unless someone remembers them. +# check-skill-mdl is a superset of the job's three skill/docs MDL steps. +# +# Left out: govulncheck (the job installs it @latest, over the network) and the +# integration jobs, which need mxbuild -- `make test-integration` runs those. +# A step added to the job belongs here too. +ci-local: build test check-tunnel-deps check-mdl check-skill-pack-js check-findings \ + check-wiki-pages check-skill-index check-test-timeouts check-skill-mdl \ + check-conformance check-migration-reference lint-go + @echo "ci-local passed (build-and-test, without govulncheck)" + # Lint Go code # # Depends on fmt-check, which VERIFIES, not on fmt, which rewrites. `go fmt ./...` From 880a2841c6e6c154baf7f9505a1b0c2427e7eb18 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 19:49:31 +0000 Subject: [PATCH 14/44] docs(skills): add the steward skill, the loop from PR opened to merged Holds this repository's conventions for driving a PR Claude opened to green: the user reviews before the PR is opened and it is merged on green by default (merge commit, expectedHeadSha, branch restarted from main afterwards); `make ci-local` before every push; the macOS tunnel-seam job that is cancelled before it gets a runner, and so gets one re-run; the conflicts this repo keeps hitting (CHANGELOG, generated indexes and tables); and how to treat the AI PR Review comment's Critical / Moderate / Minor sections. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .claude/skills/steward/SKILL.md | 121 ++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 .claude/skills/steward/SKILL.md diff --git a/.claude/skills/steward/SKILL.md b/.claude/skills/steward/SKILL.md new file mode 100644 index 000000000..76b305efe --- /dev/null +++ b/.claude/skills/steward/SKILL.md @@ -0,0 +1,121 @@ +--- +name: steward +description: Drive an mxcli pull request Claude opened to green and merge it. Covers when to open it, the local gate to pass before every push (make ci-local), which CI failure is a known runner flake, how to resolve the conflicts this repo keeps hitting, how to read the AI PR Review comment, and the merge-on-green default. Read before acting on a CI, review or conflict event on such a PR. +--- + +## Find it fast + +`sed -n ',p' SKILL.md` one section rather than reading the file; +`./mxcli syntax ` answers *how do I write this* for a fraction of either. + +| lines | section | +|---|---| +| 27-54 | Merge on green is the default | +| 55-72 | Before every push: `make ci-local` | +| 73-87 | Reading a red CI job | +| 88-103 | Conflicts this repo keeps hitting | +| 104-121 | The AI PR Review comment | + + +# Steward: Driving a PR to Merged + +This is a **contributor** skill (dev-only); it is not synced to user projects. +It holds this repository's conventions for the loop after a PR is opened. How +to make the change itself is `/mxcli-dev:fix-issue` and the CLAUDE.md working +rules; this file starts where they stop. + +## Merge on green is the default + +The user reviews a change **in the session, before the PR exists**. So: + +- Open a PR only after the user has seen the change summarized in the session + and asked for it. Never open one to get a review. +- Once it is open, merge it without asking again as soon as all of these hold + on the **current head**: + - every job of `Build, Test & Lint` (`.github/workflows/push-test.yml`) is + green, `integration-passed` included. A push to the PR branch runs that + workflow twice on the same commit, for the push and the pull_request + event; a job that passed in one run and was cancelled before it got a + runner in the other counts as green; + - GitHub reports no merge conflict; + - no human review thread or changes-requested review is open; + - no Critical item from the AI PR Review comment is unanswered (see below). +- Merge with a **merge commit** (not squash, not rebase), passing the head SHA + you checked as `expectedHeadSha`, so a push that lands in between is not + merged unseen. +- After the merge, unsubscribe from the PR, cancel its pending check-in, and + restart the working branch from the new main: + `git fetch origin main && git checkout -B origin/main`, then push. + The next PR starts from there; never stack new work on merged history. + +Ask instead of merging when a human reviewer has commented or requested +changes, or when the PR is someone else's. Never merge a red head, whatever +the reason it is red. + +## Before every push: `make ci-local` + +`make build test lint` is a subset of the gate CI applies. `make ci-local` +runs what the `build-and-test` job runs, in its order, minus `govulncheck`. +Run it before the first push and before every push that changes code, and +push only on a clean exit. It takes a while; run it in the background. + +- **Formatting:** `make fmt` is the fixer. `fmt-check` uses the toolchain's + gofmt (`go env GOROOT`), because the gofmt on `PATH` can be an older Go than + the `toolchain` line in `go.mod`, and its verdict differs from CI's. +- **Integration suites** need mxbuild, so `ci-local` does not run them. When + the change touches a package with integration tests, run that suite: + `make test-integration-`. +- **When the job gains a step,** add it to `ci-local` in the same PR. +- **A failure in code the change does not touch:** run the same target on + `origin/main`. If it fails there too, it is main's: fix it in its own PR + (it can go first), and never skip or disable the test to get past it. + +## Reading a red CI job + +The workflow is the source of truth for what runs; these are the jobs whose +failures need reading differently. + +- **`build-and-test`:** reproduce with `make ci-local`. +- **`integration ()`:** reproduce with `make test-integration-`. +- **`tunnel-seam-cross-platform (macos-latest | windows-latest)`:** the one + known runner flake. The hosted macOS queue sometimes cancels the job before + it gets a runner (the job shows runner id 0 and no step ran). That is a job + that died before any test body ran: re-run the failed jobs **once**. A test + failure in this job is real: it runs the tunnel seam's own tests on that + platform (ADR-0009). +- **`AI PR Review`** is always green: it posts a comment, it does not gate. + +## Conflicts this repo keeps hitting + +While the PR is open, merge `origin/main` into its head. Do not rebase or +force-push it. + +| File | Resolution | +|---|---| +| `CHANGELOG.md` `[Unreleased]` | Keep **both** sides: every entry is one line, so a conflict is two fixes landing at once. Afterwards check for a stray blank line or a duplicated entry. | +| A `SKILL.md` section index | Keep either side, then `make skill-index`. | +| `cmd/mxcli/skills/` | Never resolve by hand: `make sync-skills` regenerates it from `.claude/skills/mendix/`. | +| `docs-site/src/language/versions.md` tables | `make gen-migration-reference`. | +| `mdl/conformance/allowlist.txt` | Take the higher count on each conflicting line, then `make conformance-shrink`, which lowers every count to what the gate finds and never raises one. | +| Bug findings | They do not conflict: each finding is its own new file. | + +Run `make ci-local` on the merge result before pushing it. + +## The AI PR Review comment + +`ai-review.yml` posts **one PR comment** headed `## AI Code Review`, not review +threads. It has sections Critical Issues, Moderate Issues, Minor Issues, What +Looks Good and Recommendation. On a large PR the model sees only a selected +part of the diff, so a claim that something is missing may only mean that file +was outside the selection. + +| Section | How to treat it | +|---|---| +| Critical | Blocking. Verify each item against the code: fix and push it, or answer it with the path you traced and why it does not hold. | +| Moderate | A claim to verify, not an order. Fix it when a real caller reaches the failure and the fix is in proportion to it; otherwise say why not. | +| Minor | Optional. Never start a push for one. Carry the plainly correct ones into a push that already touches those files. | + +Answer it with **one** PR comment that gives the outcome for each Critical and +Moderate item. Sometimes nothing is posted at all: the workflow skips posting +when its API key is missing or the API call fails, and still reports success. +A missing comment is neither an approval nor a blocker. From 277bf8e54eff42a0ce3d79af7b9c971d435002ae Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 21:55:19 +0000 Subject: [PATCH 15/44] fix(git): doctor judges notes only once the remote's notes are fetched `git clone` never fetches refs/notes/*, so on a fresh clone or a CI checkout `git doctor` reported every commit as "without a note" and exited 1, while the same report said `[x] origin has refs/notes/mx_metadata`. `--fix` then wrote computed notes over commits the remote already annotates, and the push it recommends was rejected as non-fast-forward. Anyone who forced that push would overwrite the real notes on the remote. The remote check already reads the remote's notes commit with ls-remote; it is now passed on. If the local ref is not, or does not contain, that commit, the commits are not judged. The finding is the unfetched ref, given with the command that fetches it. `--fix` fetches it as a fast-forward only, so a note that exists only locally is never dropped, and stops if that fetch is refused. Test: TestGitDoctor_UnfetchedNotesAreNotMissing (bare remote, pushed note, fresh clone). Before the fix it fails with the reported symptom: "[ ] 9450425668 without a note" next to "[x] origin has refs/notes/mx_metadata", and --fix replaced the remote's note on HEAD. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...judges-notes-against-an-unfetched-ref.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_doctor.go | 11 +++-- cmd/mxcli/gitdoctor.go | 46 ++++++++++++++++++- cmd/mxcli/gitdoctor_test.go | 42 +++++++++++++++++ docs-site/src/tools/outside-studio-pro.md | 11 +++++ 6 files changed, 107 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-notes-against-an-unfetched-ref.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-notes-against-an-unfetched-ref.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-notes-against-an-unfetched-ref.json new file mode 100644 index 000000000..0e730ec6a --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-notes-against-an-unfetched-ref.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "`mxcli git doctor` on a fresh clone (or any CI checkout) reports every commit as `[ ] without a note` and exits 1, while the same report says `[x] origin has refs/notes/mx_metadata`; `--fix` then writes computed notes over commits the remote already annotates, and the `git push origin refs/notes/mx_metadata` it tells you to run is rejected as non-fast-forward", "cause": "doctorNotes judged each commit against the LOCAL refs/notes/mx_metadata only, and nothing fetched it: git clone and a plain git fetch never fetch refs/notes/*. The remote check had already run `git ls-remote refs/notes/mx_metadata` and printed the tip, but that result was not passed on, so an empty local ref read as 'no commit has a note'", "file": "`cmd/mxcli/gitdoctor.go` (`doctorRun` passes the remote's notes commit to `doctorNotes`; `hasNotesCommit`: the local ref is, or descends from, that commit); test `TestGitDoctor_UnfetchedNotesAreNotMissing` in `cmd/mxcli/gitdoctor_test.go`", "insight": "**Every existing notes test built its repository with `git init`, never `git clone`**, so the local notes ref was always the only one there was; the defect lives exactly in the gap between a repository that writes notes and one that receives them. A fixture with a bare remote, a pushed note and a fresh clone reproduced it on the first run. **Do not judge, and do not write, against a ref that is behind the remote**: Studio Pro's own fetch force-replaces the local notes ref with the remote's, so the remote's notes are the ones that count; a note written over a remote-annotated commit is either refused on push or thrown away on Studio Pro's next fetch. **Fetch as a fast-forward only** (`git fetch refs/notes/mx_metadata:refs/notes/mx_metadata`, no `+`): it never discards a note that exists only locally, which a force-fetch would; when the local ref has diverged the fetch is refused and the doctor says so instead of writing. Report mode stays read-only: it names the unfetched ref and the command rather than fetching", "refs": ["ako/mxcli#1670", "mendixlabs/mxcli#1669"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 12a35f81c..e97d561d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. Docs: "Working Outside Studio Pro" › *Auditing a repository*. +- **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_doctor.go b/cmd/mxcli/cmd_git_doctor.go index 1e17d23f3..486c56323 100644 --- a/cmd/mxcli/cmd_git_doctor.go +++ b/cmd/mxcli/cmd_git_doctor.go @@ -29,16 +29,21 @@ Checked: (git@host:org/app.git or ssh://…) is rewritten to HTTPS by --fix. Other non-HTTPS URLs are reported without being changed. 3. The remote's refs/notes/mx_metadata. Absent or unreachable is a hint: - push the notes ref together with the branch. + push the notes ref together with the branch. When the remote has notes + the local ref does not hold (git clone never fetches refs/notes/*), the + commits are not judged: that is the finding, and --fix fetches the ref, + as a fast-forward only. 4. Every commit's mx_metadata note. Missing notes make Studio Pro show "(unknown)"; a malformed note it cannot read at all. --fix computes real notes from the model (with -p), and falls back to a template note with a --version ModelerVersion where the model at a commit cannot be read. Notes Studio Pro wrote are never touched. -Exits 1 while an issue remains, so the audit can gate CI. --fix does not push: -push the branch and the notes ref together yourself. +Exits 1 while an issue remains, so the audit can gate CI; fetch the notes +first there. --fix does not push: push the branch and the notes ref together +yourself. + git fetch refs/notes/mx_metadata:refs/notes/mx_metadata git push refs/notes/mx_metadata`, Example: ` mxcli git doctor mxcli git doctor -p app.mpr --fix diff --git a/cmd/mxcli/gitdoctor.go b/cmd/mxcli/gitdoctor.go index 48f045728..d01dfbaca 100644 --- a/cmd/mxcli/gitdoctor.go +++ b/cmd/mxcli/gitdoctor.go @@ -76,6 +76,7 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W fmt.Fprintln(out, "Remote") remote := detectDoctorRemote(root, remoteOverride) + remoteNotes := "" // the remote's notes commit, when it has one and could be reached switch { case remote == "": fmt.Fprintln(out, " [i] no remote configured; skipping remote checks") @@ -112,12 +113,15 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W fmt.Fprintf(out, " [i] could not reach %s to check %s\n", remote, mxMetadataRef) case strings.Contains(ls, mxMetadataRef): fmt.Fprintf(out, " [x] %s has %s\n", remote, mxMetadataRef) + if f := strings.Fields(ls); len(f) > 0 { + remoteNotes = f[0] + } default: fmt.Fprintf(out, " [i] %s has no %s yet; push it together with the branch\n", remote, mxMetadataRef) } } - fn, fx, err := doctorNotes(root, mprPath, version, fix, out) + fn, fx, err := doctorNotes(root, mprPath, version, remote, remoteNotes, fix, out) if err != nil { return err } @@ -143,12 +147,35 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W // doctorNotes reports (and with fix, repairs) missing and malformed notes over // the whole current history. It returns how many it found and how many it fixed. -func doctorNotes(root, mprPath, version string, fix bool, out io.Writer) (found, fixed int, err error) { +// +// remoteNotes is the remote's notes commit ("" when it has none or could not +// be reached). Those are the notes Studio Pro reads: its fetch force-replaces +// the local ref with them. A local ref that does not contain that commit — a +// fresh clone, since git clone never fetches refs/notes/* — cannot be judged: +// every commit the remote annotates would look unannotated, and a note written +// over it could not be pushed (non-fast-forward). So the commits are not +// judged until the ref is fetched; --fix fetches it, as a fast-forward only, +// which never drops a note that exists only locally. +func doctorNotes(root, mprPath, version, remote, remoteNotes string, fix bool, out io.Writer) (found, fixed int, err error) { fmt.Fprintln(out, "mx_metadata notes") if _, err := gitOut(root, "rev-parse", "--verify", "--quiet", "HEAD"); err != nil { fmt.Fprintln(out, " [i] no commits yet") return 0, 0, nil } + if remoteNotes != "" && !hasNotesCommit(root, remoteNotes) { + found++ + fmt.Fprintf(out, " [ ] %s from %s is not fetched; the notes cannot be judged without it\n", mxMetadataRef, remote) + if !fix { + fmt.Fprintf(out, " git fetch %s %s:%s\n", remote, mxMetadataRef, mxMetadataRef) + return found, 0, nil + } + if _, err := gitOut(root, "fetch", "--quiet", remote, mxMetadataRef+":"+mxMetadataRef); err != nil { + fmt.Fprintf(out, " could not fetch it as a fast-forward (%v); if the local ref has notes the remote lacks, push or reconcile them first\n", err) + return found, 0, nil + } + fixed++ + fmt.Fprintf(out, " fetched from %s\n", remote) + } revs := strings.Fields(gitOutOr(root, "rev-list", "--reverse", "HEAD")) noted := map[string]bool{} for _, line := range strings.Split(strings.TrimRight(gitOutOr(root, "notes", "--ref="+mxMetadataRef, "list"), "\n"), "\n") { @@ -316,6 +343,21 @@ func sshToHTTPS(url string) (host, p string, ok bool) { return "", "", false } +// hasNotesCommit reports whether the local notes ref is, or contains, the +// notes commit c. False when the ref is absent or c is not in its history +// (including when c's object was never fetched). +func hasNotesCommit(root, c string) bool { + local, err := gitOut(root, "rev-parse", "--verify", "--quiet", mxMetadataRef) + if err != nil { + return false + } + if strings.TrimSpace(local) == c { + return true + } + _, err = gitOut(root, "merge-base", "--is-ancestor", c, strings.TrimSpace(local)) + return err == nil +} + func gitNoteText(root, c string) string { out, err := gitOut(root, "notes", "--ref="+mxMetadataRef, "show", c) if err != nil { diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index fb2cf56c1..be8a948d4 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -218,6 +218,48 @@ func TestGitDoctor_FixKeepsRealNotes(t *testing.T) { } } +// A fresh clone has no refs/notes/* (git clone never fetches them), while the +// remote carries the notes Studio Pro wrote. Judged against the empty local +// ref, every commit looked unannotated, and --fix wrote notes over the +// remote's that it then refused to accept (non-fast-forward). +func TestGitDoctor_UnfetchedNotesAreNotMissing(t *testing.T) { + src, _ := doctorNoteRepo(t) + doctorEnv(t, src) + gitIn(t, src, "notes", "--ref="+mxMetadataRef, "add", "-f", "-m", realStudioProNote, "HEAD") + remote := t.TempDir() + gitIn(t, remote, "init", "-q", "--bare") + gitIn(t, src, "remote", "add", "origin", remote) + gitIn(t, src, "push", "-q", "origin", "feature", mxMetadataRef) + + clone := filepath.Join(t.TempDir(), "clone") + gitIn(t, filepath.Dir(clone), "clone", "-q", "-b", "feature", remote, clone) + for _, kv := range mxConfigKeys { + gitIn(t, clone, "config", "--local", kv[0], kv[1]) + } + mprPath := filepath.Join(clone, "App.mpr") + head := strings.TrimSpace(gitOutOr(clone, "rev-parse", "HEAD")) + + var out bytes.Buffer + err := doctorRun(clone, mprPath, "", "", false, &out) + if strings.Contains(out.String(), shortCommit(head)+" without a note") { + t.Errorf("HEAD has a note on the remote; an unfetched notes ref must not report it missing:\n%s", out.String()) + } + if !errors.Is(err, errDoctorIssuesRemain) || !strings.Contains(out.String(), "git fetch origin "+mxMetadataRef) { + t.Errorf("the unfetched notes ref must be the finding, with the command that fetches it: %v\n%s", err, out.String()) + } + + out.Reset() + if err := doctorRun(clone, mprPath, "", "", true, &out); err != nil { + t.Fatalf("--fix must fetch the notes and annotate the rest: %v\n%s", err, out.String()) + } + if got := strings.TrimSpace(noteOn(t, clone, "HEAD")); got != realStudioProNote { + t.Fatalf("--fix replaced the remote's note on HEAD:\n got %s\nwant %s\n%s", got, realStudioProNote, out.String()) + } + if _, err := gitOut(clone, "push", "origin", "feature", mxMetadataRef); err != nil { + t.Errorf("the notes --fix leaves must push as a fast-forward: %v\n%s", err, out.String()) + } +} + func TestGitDoctor_TemplateWithoutProjectUsesVersionFlag(t *testing.T) { dir := t.TempDir() doctorEnv(t, dir) diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 9cf80d53a..6c38a75b9 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -227,11 +227,22 @@ mxcli git doctor --fix --version 10.21.0 | local git config: `core.autocrlf=False` (a checkout must not rewrite line endings inside `.mxunit` files), `mendix.commits-since-gc`, `mendix.lineEndingResetDone` | sets them | | the remote URL: Studio Pro can only use HTTPS | rewrites SSH (`git@host:org/app.git`, `ssh://…`) to HTTPS; other non-HTTPS URLs are reported unchanged | | the remote's `refs/notes/mx_metadata` | reported only — push the notes ref together with the branch | +| the local `refs/notes/mx_metadata` holds the remote's notes: `git clone` and a plain `git fetch` never fetch them, so a fresh clone or a CI checkout has none | fetches them, as a fast-forward only; refused when the local ref has notes the remote does not | | each commit on the checked-out branch and its mx_metadata note: missing notes make Studio Pro show `(unknown)`; a malformed note it cannot read at all | computes real notes from the model (`-p`), falling back to a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); when no version is known the newest valid note in history supplies it | Notes Studio Pro wrote are never touched, with or without `--fix`. `--fix` does not push. +The commits are judged only once the local notes ref holds the remote's. Until +then the report names the unfetched ref and the command that fetches it, rather +than calling every commit the remote annotates unannotated. In CI, fetch the +notes before the audit: + +```bash +git fetch origin refs/notes/mx_metadata:refs/notes/mx_metadata +mxcli git doctor -p app.mpr +``` + ## Commit metadata: `mxcli git note` Studio Pro attaches a git note under `refs/notes/mx_metadata` to every commit From 61af7ee22785aa61e32bd3e10232ed3b8824a8ed Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 21:57:11 +0000 Subject: [PATCH 16/44] fix(git): doctor judges the configured remote URL, not the insteadOf rewrite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `git doctor` read the remote with `git remote get-url`, which applies url..insteadOf. With a global https -> git@ rule (a common developer setup), an HTTPS remote read as SSH and the doctor exited 1. `--fix` then set the configured URL to HTTPS on every run, which the rule rewrote straight back, and dropped the URL's `.git` suffix along the way. ako/mxcli#1765 isolated the tests from this and left the product question open; this answers it. The check now reads `git config --get remote..url`, the value --fix actually writes. An SSH remote is still found and rewritten when a rule makes it read as HTTPS. When the effective URL differs, the report gives it as an [i] line naming the insteadOf rule, which --fix leaves alone. It is not counted as an issue: the rule applies to every git command on the machine, and changing the remote cannot undo it. Tests: TestGitDoctor_InsteadOfRewriteIsNotAnSSHRemote and TestGitDoctor_ConfiguredSSHRemoteFoundBehindRewrite. Both use a rule in the test repository's own config, on a host nobody rewrites. Before the fix: "origin: git@git.example.com:org/app.git — SSH; Studio Pro needs HTTPS", --fix changed the configured URL to https://git.example.com/org/app, and the configured SSH remote was missed. The two existing SSH tests also pass now in an environment that rewrites github.com URLs. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...es-the-insteadof-rewritten-remote-url.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_doctor.go | 8 +-- cmd/mxcli/gitdoctor.go | 9 +++- cmd/mxcli/gitdoctor_test.go | 50 +++++++++++++++++++ docs-site/src/tools/outside-studio-pro.md | 2 +- 6 files changed, 66 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-the-insteadof-rewritten-remote-url.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-the-insteadof-rewritten-remote-url.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-the-insteadof-rewritten-remote-url.json new file mode 100644 index 000000000..608387c55 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-doctor-judges-the-insteadof-rewritten-remote-url.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "With a global `url.git@github.com:.insteadOf https://github.com/` rule (a common developer setup), `mxcli git doctor` reports an HTTPS remote as `origin: git@github.com:org/app.git — SSH; Studio Pro needs HTTPS`, exits 1, and `--fix` \"rewrites\" the configured URL to HTTPS on every run (dropping its `.git` suffix) without changing anything git uses", "cause": "The remote check read `git remote get-url`, which applies url..insteadOf, and --fix set `remote..url`, which the rule rewrites again on the next read, so the check could never pass. The same read made the two SSH tests depend on the developer's and the sandbox's git config (ako/mxcli#1765 isolated the tests; the product question stayed open)", "file": "`cmd/mxcli/gitdoctor.go` (`doctorRun`: judge `git config --get remote..url`; report a differing `git remote get-url` as an [i] insteadOf line); tests `TestGitDoctor_InsteadOfRewriteIsNotAnSSHRemote`, `TestGitDoctor_ConfiguredSSHRemoteFoundBehindRewrite` in `cmd/mxcli/gitdoctor_test.go`", "insight": "**A check and its fix must read and write the same thing.** `remote get-url` is the effective URL, `remote set-url` writes the configured one; with a rewrite in between, a fix can never make its own check pass, and a loop like that is the tell. Judge what you can change (the configured URL) and report what you cannot (the rule, which applies to every git command on the machine). **Put the rule in the test repository's own config and use a host nobody rewrites** (git.example.com): the test then shows the behaviour without inheriting the developer's or the CI sandbox's rewrites of github.com, which point in opposite directions. Set GIT_SSH_COMMAND=false when the effective URL is SSH, or ls-remote reaches for the network", "refs": ["ako/mxcli#1670", "ako/mxcli#1765"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index e97d561d1..d13bae0fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. +- **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_doctor.go b/cmd/mxcli/cmd_git_doctor.go index 486c56323..06fb78f18 100644 --- a/cmd/mxcli/cmd_git_doctor.go +++ b/cmd/mxcli/cmd_git_doctor.go @@ -25,9 +25,11 @@ Checked: 1. Local git config: core.autocrlf=False (a checkout must not rewrite line endings inside .mxunit files), mendix.commits-since-gc and mendix.lineEndingResetDone (Studio Pro's own bookkeeping). --fix sets them. - 2. The remote URL. Studio Pro can only use HTTPS; an SSH remote - (git@host:org/app.git or ssh://…) is rewritten to HTTPS by --fix. Other - non-HTTPS URLs are reported without being changed. + 2. The remote URL, as configured. Studio Pro can only use HTTPS; an SSH + remote (git@host:org/app.git or ssh://…) is rewritten to HTTPS by --fix. + Other non-HTTPS URLs are reported without being changed. A + url..insteadOf rule in your git config that makes git use another + URL is reported too; --fix leaves it alone. 3. The remote's refs/notes/mx_metadata. Absent or unreachable is a hint: push the notes ref together with the branch. When the remote has notes the local ref does not hold (git clone never fetches refs/notes/*), the diff --git a/cmd/mxcli/gitdoctor.go b/cmd/mxcli/gitdoctor.go index d01dfbaca..0d106e9f6 100644 --- a/cmd/mxcli/gitdoctor.go +++ b/cmd/mxcli/gitdoctor.go @@ -81,7 +81,11 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W case remote == "": fmt.Fprintln(out, " [i] no remote configured; skipping remote checks") default: - urlRaw, urlErr := gitOut(root, "remote", "get-url", remote) + // The configured URL, not `git remote get-url`: that one applies + // url..insteadOf, so a rewrite in the user's git config (https -> + // git@ is a common one) made an HTTPS remote read as SSH, and --fix set + // the configured URL on every run without changing what git uses. + urlRaw, urlErr := gitOut(root, "config", "--get", "remote."+remote+".url") if urlErr != nil { found++ fmt.Fprintf(out, " [ ] remote %s does not exist (git remote -v lists the names)\n", remote) @@ -106,6 +110,9 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W } else { fmt.Fprintf(out, " [i] %s: %s (not HTTPS; Studio Pro can only use HTTPS remotes)\n", remote, url) } + if eff, err := gitOut(root, "remote", "get-url", remote); err == nil && strings.TrimSpace(eff) != url { + fmt.Fprintf(out, " [i] a url..insteadOf rule in your git config makes git use %s instead; --fix leaves the rule alone\n", strings.TrimSpace(eff)) + } // Absent or unreachable notes ref is a hint, not an issue. ls, err := gitOut(root, "ls-remote", remote, mxMetadataRef) switch { diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index be8a948d4..1a3b49e3e 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -143,6 +143,56 @@ func TestGitDoctor_SSHURLFormRewrittenByFix(t *testing.T) { } } +// A url..insteadOf rule rewrites the URL every git command uses, not the +// URL the repository is configured with. The doctor judged the rewritten one: +// with the common https -> git@ rule in ~/.gitconfig an HTTPS remote read as +// SSH, and --fix "repaired" the configured URL on every run. +func TestGitDoctor_InsteadOfRewriteIsNotAnSSHRemote(t *testing.T) { + dir := t.TempDir() + doctorEnv(t, dir) + t.Setenv("GIT_SSH_COMMAND", "false") // the rewritten URL is SSH: never reach the network + gitIn(t, dir, "init", "-q", "-b", "main") + for _, kv := range mxConfigKeys { + gitIn(t, dir, "config", "--local", kv[0], kv[1]) + } + gitIn(t, dir, "config", "--local", "url.git@git.example.com:.insteadOf", "https://git.example.com/") + gitIn(t, dir, "remote", "add", "origin", "https://git.example.com/org/app.git") + + var out bytes.Buffer + if err := doctorRun(dir, "", "", "", false, &out); err != nil { + t.Errorf("a configured HTTPS remote is not an issue, whatever a rewrite makes of it: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "insteadOf") { + t.Errorf("the report must explain the rewrite:\n%s", out.String()) + } + + out.Reset() + if err := doctorRun(dir, "", "", "", true, &out); err != nil { + t.Errorf("--fix: %v\n%s", err, out.String()) + } + if got := configLocal(t, dir, "remote.origin.url"); got != "https://git.example.com/org/app.git" { + t.Errorf("--fix changed the configured URL to %q:\n%s", got, out.String()) + } +} + +// The configured URL is what --fix rewrites, so an SSH remote is still found +// when a rule rewrites it to HTTPS (as hosted sandboxes do). +func TestGitDoctor_ConfiguredSSHRemoteFoundBehindRewrite(t *testing.T) { + dir := t.TempDir() + doctorEnv(t, dir) + gitIn(t, dir, "init", "-q", "-b", "main") + gitIn(t, dir, "config", "--local", "url.https://git.example.com/.insteadOf", "git@git.example.com:") + gitIn(t, dir, "remote", "add", "origin", "git@git.example.com:org/app.git") + + var out bytes.Buffer + if err := doctorRun(dir, "", "", "", true, &out); err != nil { + t.Fatalf("--fix: %v\n%s", err, out.String()) + } + if got := configLocal(t, dir, "remote.origin.url"); got != "https://git.example.com/org/app" { + t.Errorf("configured URL = %q, want https://git.example.com/org/app\n%s", got, out.String()) + } +} + // --- doctor: mx_metadata notes ---------------------------------------------- // doctorNoteRepo makes a two-commit repository holding an MPR v2 project: diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 6c38a75b9..e48b61ff9 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -225,7 +225,7 @@ mxcli git doctor --fix --version 10.21.0 | Check | `--fix` | |-------|---------| | local git config: `core.autocrlf=False` (a checkout must not rewrite line endings inside `.mxunit` files), `mendix.commits-since-gc`, `mendix.lineEndingResetDone` | sets them | -| the remote URL: Studio Pro can only use HTTPS | rewrites SSH (`git@host:org/app.git`, `ssh://…`) to HTTPS; other non-HTTPS URLs are reported unchanged | +| the remote URL as configured: Studio Pro can only use HTTPS | rewrites SSH (`git@host:org/app.git`, `ssh://…`) to HTTPS; other non-HTTPS URLs are reported unchanged. A `url..insteadOf` rule in your git config that makes git use another URL is reported, and left alone | | the remote's `refs/notes/mx_metadata` | reported only — push the notes ref together with the branch | | the local `refs/notes/mx_metadata` holds the remote's notes: `git clone` and a plain `git fetch` never fetch them, so a fresh clone or a CI checkout has none | fetches them, as a fast-forward only; refused when the local ref has notes the remote does not | | each commit on the checked-out branch and its mx_metadata note: missing notes make Studio Pro show `(unknown)`; a malformed note it cannot read at all | computes real notes from the model (`-p`), falling back to a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); when no version is known the newest valid note in history supplies it | From eac6fbfb02b34e14aaaa9fbc894817bedd6da749 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 21:59:07 +0000 Subject: [PATCH 17/44] fix(git): commit --dry-run leaves what the user staged as it was `mxcli git commit --dry-run` staged everything in the real index to report it, then undid that with `git reset -q`. A reset goes back to HEAD, not to the state before the run, so anything the user had staged was unstaged. The docs promise that a dry run writes nothing. A dry run now stages into a temporary copy of the index (GIT_INDEX_FILE), so the real index is never touched: staged changes, stat data and intent-to-add entries all stay as they were. A real commit still uses the real index, as before. `gitOutInEnv` is the git helper with extra environment. Test: TestGitCommit_DryRunKeepsTheUsersStaging. Before the fix: `staged after --dry-run = "", want exactly what the user had staged (staged.txt)`. The existing dry-run test started from an empty index, where the bug and the correct behaviour look the same. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...it-dry-run-unstages-the-users-staging.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_commit.go | 70 ++++++++++++++++--- cmd/mxcli/gitdoctor_test.go | 28 ++++++++ cmd/mxcli/gitnote.go | 8 ++- docs-site/src/tools/outside-studio-pro.md | 3 +- 6 files changed, 99 insertions(+), 13 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-dry-run-unstages-the-users-staging.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-dry-run-unstages-the-users-staging.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-dry-run-unstages-the-users-staging.json new file mode 100644 index 000000000..80af6632b --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-dry-run-unstages-the-users-staging.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "`mxcli git commit -m … -p app.mpr --dry-run` unstages everything the user had staged: a file added with `git add` before the dry run is no longer staged afterwards (`git diff --cached` is empty), although the command committed nothing and documents that a dry run writes nothing", "cause": "The dry run staged everything in the real index (`git add -A`) to report it, then undid that with a plain `git reset -q`, which resets the index to HEAD rather than to what it was before the run", "file": "`cmd/mxcli/cmd_git_commit.go` (`commitRun`: a dry run stages into a temporary copy of the index through GIT_INDEX_FILE, `indexCopy`); `cmd/mxcli/gitnote.go` (`gitOutInEnv`); test `TestGitCommit_DryRunKeepsTheUsersStaging` in `cmd/mxcli/gitdoctor_test.go`", "insight": "**The existing dry-run test started from an empty index**, so 'nothing staged afterwards' was both the correct result and the bug's result; a test of an undo has to start from a state the undo can get wrong. **Do not undo, isolate**: staging into a GIT_INDEX_FILE copy leaves the real index untouched byte for byte (stat data and intent-to-add entries included), which no reset or write-tree/read-tree round trip quite does. An empty file is not a valid index, so with no index yet the temporary path is left free for git to create", "refs": ["ako/mxcli#1670", "mendixlabs/mxcli#1669"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index d13bae0fb..22b0ac578 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. +- **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index cbdb52cbe..3db953617 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -60,7 +60,7 @@ func init() { gitCommitCmd.Flags().StringP("project", "p", "", "path to the Mendix project (.mpr)") _ = gitCommitCmd.MarkFlagRequired("project") gitCommitCmd.Flags().Bool("push", false, "push the branch and the notes ref to their remote afterwards") - gitCommitCmd.Flags().Bool("dry-run", false, "stage and report, then unstage; commit nothing") + gitCommitCmd.Flags().Bool("dry-run", false, "report what would be committed; commit nothing and leave your staging as it is") gitCmd.AddCommand(gitCommitCmd) } @@ -115,12 +115,25 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error } } - // 2. Stage everything, then unstage what Studio Pro never commits. - if _, err := gitOutIn(root, nil, "add", "-A"); err != nil { + // 2. Stage everything, then unstage what Studio Pro never commits. A dry + // run stages into a copy of the index: resetting the real one afterwards + // unstaged whatever the user had staged before running it. + var env []string + if dryRun { + idx, cleanup, err := indexCopy(root) + if err != nil { + return fmt.Errorf("copy the index for the dry run: %w", err) + } + defer cleanup() + env = []string{"GIT_INDEX_FILE=" + idx} + } + git := func(args ...string) (string, error) { return gitOutInEnv(root, env, nil, args...) } + if _, err := git("add", "-A"); err != nil { return fmt.Errorf("stage changes: %w", err) } // -z: paths come back NUL-separated, so spaces and quotes stay inside one path. - staged := nulFields(gitOutOr(root, "diff", "--cached", "--name-only", "-z")) + cached, _ := git("diff", "--cached", "--name-only", "-z") + staged := nulFields(cached) var junk []string for _, f := range staged { if isJunkPath(f) { @@ -128,8 +141,7 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error } } if len(junk) > 0 { - args := append([]string{"reset", "-q", "--"}, junk...) - if _, err := gitOutIn(root, nil, args...); err != nil { + if _, err := git(append([]string{"reset", "-q", "--"}, junk...)...); err != nil { return fmt.Errorf("unstage files Studio Pro never commits: %w", err) } fmt.Fprintf(out, "Left out %d file(s) Studio Pro never commits (project-settings.user.json, *.mpr.lock, caches).\n", len(junk)) @@ -151,10 +163,7 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error } if dryRun { - if _, err := gitOutIn(root, nil, "reset", "-q"); err != nil { - return fmt.Errorf("unstage: %w", err) - } - fmt.Fprintln(out, "Dry run: staged nothing permanently, committed nothing.") + fmt.Fprintln(out, "Dry run: committed nothing; what you had staged is unchanged.") return nil } @@ -185,6 +194,47 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error return nil } +// indexCopy copies the repository's index to a temporary file, for a +// GIT_INDEX_FILE that a dry run can stage into without touching the real one. +// With no index yet, the path is left free for git to create it. +func indexCopy(root string) (string, func(), error) { + p, err := gitOut(root, "rev-parse", "--git-path", "index") + if err != nil { + return "", nil, err + } + src := strings.TrimSpace(p) + if !filepath.IsAbs(src) { + src = filepath.Join(root, src) + } + f, err := os.CreateTemp("", "mxcli-dry-run-index-*") + if err != nil { + return "", nil, err + } + tmp := f.Name() + cleanup := func() { _ = os.Remove(tmp) } + data, err := os.ReadFile(src) + switch { + case errors.Is(err, os.ErrNotExist): + f.Close() + cleanup() // an empty file is not a valid index + return tmp, cleanup, nil + case err != nil: + f.Close() + cleanup() + return "", nil, err + } + if _, err := f.Write(data); err != nil { + f.Close() + cleanup() + return "", nil, err + } + if err := f.Close(); err != nil { + cleanup() + return "", nil, err + } + return tmp, cleanup, nil +} + // nulFields splits git's -z output (NUL-separated, no quoting) into paths. func nulFields(s string) []string { var out []string diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index 1a3b49e3e..edb2b38ec 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -458,6 +458,34 @@ func TestGitCommit_RefusedWhileStudioProOpen(t *testing.T) { } } +// --dry-run staged everything to report it and then reset the whole index, so +// whatever the user had staged before running it was unstaged. +func TestGitCommit_DryRunKeepsTheUsersStaging(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + for _, name := range []string{"staged.txt", "unstaged.txt"} { + if err := os.WriteFile(filepath.Join(dir, name), []byte(name), 0o644); err != nil { + t.Fatal(err) + } + } + gitIn(t, dir, "add", "staged.txt") + + var out bytes.Buffer + if err := commitRun(mprPath, "dry", false, true, &out); err != nil { + t.Fatalf("dry run failed: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "unstaged.txt") { + t.Errorf("the dry run must still report every file it would commit:\n%s", out.String()) + } + staged, err := gitOut(dir, "diff", "--cached", "--name-only") + if err != nil { + t.Fatal(err) + } + if got := strings.TrimSpace(staged); got != "staged.txt" { + t.Errorf("staged after --dry-run = %q, want exactly what the user had staged (staged.txt)", got) + } +} + func TestGitCommit_DryRunCommitsNothing(t *testing.T) { dir, mprPath := doctorNoteRepo(t) doctorEnv(t, dir) diff --git a/cmd/mxcli/gitnote.go b/cmd/mxcli/gitnote.go index 24a5934d6..b9e55e1bd 100644 --- a/cmd/mxcli/gitnote.go +++ b/cmd/mxcli/gitnote.go @@ -366,8 +366,14 @@ func gitOut(dir string, args ...string) (string, error) { } func gitOutIn(dir string, stdin []byte, args ...string) (string, error) { + return gitOutInEnv(dir, nil, stdin, args...) +} + +// gitOutInEnv is gitOutIn with extra environment entries, such as the +// GIT_INDEX_FILE a dry run stages into. +func gitOutInEnv(dir string, env []string, stdin []byte, args ...string) (string, error) { cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) - cmd.Env = append(os.Environ(), "GIT_TERMINAL_PROMPT=0", "GIT_PAGER=cat") + cmd.Env = append(append(os.Environ(), "GIT_TERMINAL_PROMPT=0", "GIT_PAGER=cat"), env...) if stdin != nil { cmd.Stdin = bytes.NewReader(stdin) } diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index e48b61ff9..2da3867e1 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -208,7 +208,8 @@ The commit is refused while Studio Pro has the project open (the `.mpr.lock` beside it), and during an unresolved merge — `git add -A` would otherwise commit the conflict markers. `--dry-run` verifies and reports (including ContentsHash drift, which it does not repair — a dry run writes nothing), -stages, reports what it would commit, and unstages again without committing. +and reports what it would commit without committing. It stages into a copy of +the index, so whatever you had staged is left exactly as it was. ## Auditing a repository: `mxcli git doctor` From 6a7934936e209ad31b3ade3809b0859761a09574 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 22:01:11 +0000 Subject: [PATCH 18/44] fix(git): commit leaves out only the project's own build folders `mxcli git commit` dropped any path with a segment named `.mendix-cache`, `deployment`, `theme-cache` or `.mxcli`, at any depth. So `resources/deployment/endpoints.xml` was silently left out of the commit. The only trace was a count in "Left out 1 file(s) Studio Pro never commits". Those folders are the project's own: Studio Pro, mxbuild and mxcli create them directly beside the .mpr. isJunkPath now matches them there only. When the project sits in a subfolder of the repository, that subfolder is the anchor, taken from gitNoteRepo.MPRPath. The lock, backup and user-settings file names still match at any level. Tests: TestGitCommit_NestedFolderNamedLikeJunkIsCommitted and TestGitCommit_JunkAnchoredAtProjectInSubfolder. Before the fix both fail: "resources/deployment/endpoints.xml is not the project's own folder and must be committed", and the same for javascriptsource/theme-cache/, app/resources/deployment/ and tools/deployment/. The project's own deployment/ and theme-cache/ stay out, as the existing spaced-path test also checks. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82tNcDLefAskVSzcKPm --- ...sted-folders-named-like-build-folders.json | 1 + CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_commit.go | 37 ++++++--- cmd/mxcli/gitdoctor_test.go | 79 +++++++++++++++++++ docs-site/src/tools/outside-studio-pro.md | 6 +- 5 files changed, 112 insertions(+), 13 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-drops-nested-folders-named-like-build-folders.json diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-drops-nested-folders-named-like-build-folders.json b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-drops-nested-folders-named-like-build-folders.json new file mode 100644 index 000000000..b6fbea8c4 --- /dev/null +++ b/.claude/skills/fix-issue/findings/cmd-mxcli/2026-10-28-git-commit-drops-nested-folders-named-like-build-folders.json @@ -0,0 +1 @@ +{"area": "cmd/mxcli", "date": "2026-10-28", "symptom": "`mxcli git commit` silently leaves a user file out of the commit when any folder on its path is named like a build folder: `resources/deployment/endpoints.xml` (or `javascriptsource/theme-cache/x.js`) stays untracked after the commit, with only `Left out 1 file(s) Studio Pro never commits (project-settings.user.json, *.mpr.lock, caches)` to show for it", "cause": "isJunkPath matched `.mendix-cache`, `deployment`, `theme-cache` and `.mxcli` on every path segment, so any folder of that name at any depth was dropped. Those are the project's own folders, the ones Studio Pro, mxbuild and mxcli create directly beside the .mpr; the file-name rules (lock, backup, user settings) are the ones that rightly match at any level", "file": "`cmd/mxcli/cmd_git_commit.go` (`isJunkPath(rel, projDir)`, projDir from `path.Dir(gitNoteRepo.MPRPath)`, which already resolves the project folder inside the repository, symlinks included); tests `TestGitCommit_NestedFolderNamedLikeJunkIsCommitted`, `TestGitCommit_JunkAnchoredAtProjectInSubfolder` in `cmd/mxcli/gitdoctor_test.go`", "insight": "**Anchor a path rule where the thing it names is created**: the build and cache folders only ever appear beside the .mpr, and a rule that matches the name at every depth also catches user folders that share it. **The report that something was left out did not name it**, which is what made the drop silent; a nested-folder fixture next to the root one (the existing test only had root `deployment/`) is the control that tells the two apart. Test the project-in-a-subfolder layout too: the anchor is the project's folder, not the repository root, and the same names outside the project are not its own", "refs": ["ako/mxcli#1670", "mendixlabs/mxcli#1669"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 22b0ac578..3ff88359a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. +- **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's own `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/` beside the `.mpr` — a folder of the same name further down, such as `resources/deployment/`, is committed) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index 3db953617..3942c276c 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -32,8 +32,9 @@ var gitCommitCmd = &cobra.Command{ rewritten from the file, which is the source of truth; MISSING or ORPHAN entries have no automatic repair and refuse the commit; 2. everything is staged, then the files Studio Pro never commits are - unstaged again: project-settings.user.json, *.mpr.lock, *.mpr.bak, - .mendix-cache/, deployment/, theme-cache/, .mxcli/; + unstaged again: project-settings.user.json, *.mpr.lock, *.mpr.bak, and + the project's own .mendix-cache/, deployment/, theme-cache/ and .mxcli/ + beside the .mpr (a folder of the same name further down is committed); 3. git commit runs with your message and your git identity; 4. the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check it and show what changed. @@ -134,9 +135,10 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error // -z: paths come back NUL-separated, so spaces and quotes stay inside one path. cached, _ := git("diff", "--cached", "--name-only", "-z") staged := nulFields(cached) + projDir := path.Dir(repo.MPRPath) // the project's folder in the repository; "." at its root var junk []string for _, f := range staged { - if isJunkPath(f) { + if isJunkPath(f, projDir) { junk = append(junk, f) } } @@ -247,9 +249,14 @@ func nulFields(s string) []string { } // isJunkPath reports whether a repo-relative path is one of the files Studio -// Pro never commits. Matched on any path level and without case, the way -// Studio Pro matches its own lock file (whose .gitignore entry is lowercase). -func isJunkPath(rel string) bool { +// Pro never commits. The files match by name at any level and without case, +// the way Studio Pro matches its own lock file (whose .gitignore entry is +// lowercase). The folders are the project's own, the ones Studio Pro, mxbuild +// and mxcli create directly in projDir (the project's folder relative to the +// repository root, "." at the root): matched at any depth, a user folder of +// the same name, resources/deployment/ say, was silently left out of the +// commit. +func isJunkPath(rel, projDir string) bool { f := filepath.ToSlash(rel) base := path.Base(f) switch { @@ -258,11 +265,21 @@ func isJunkPath(rel string) bool { case strings.HasSuffix(strings.ToLower(base), ".mpr.lock"), strings.HasSuffix(strings.ToLower(base), ".mpr.bak"): return true } - for _, seg := range strings.Split(f, "/") { - switch strings.ToLower(seg) { - case ".mendix-cache", "deployment", "theme-cache", ".mxcli": - return true + inProject := f + if projDir != "." && projDir != "" { + prefix := projDir + "/" + if len(f) <= len(prefix) || !strings.EqualFold(f[:len(prefix)], prefix) { + return false } + inProject = f[len(prefix):] + } + folder, rest, inFolder := strings.Cut(inProject, "/") + if !inFolder || rest == "" { + return false + } + switch strings.ToLower(folder) { + case ".mendix-cache", "deployment", "theme-cache", ".mxcli": + return true } return false } diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index edb2b38ec..5457a1b31 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -592,6 +592,85 @@ func TestGitCommit_SpacedPathsStagedWholeAndJunkLeft(t *testing.T) { } } +// writeFiles creates each repo-relative path with its parent folders. +func writeFiles(t *testing.T, dir string, paths ...string) { + t.Helper() + for _, p := range paths { + full := filepath.Join(dir, filepath.FromSlash(p)) + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(full, []byte(p), 0o644); err != nil { + t.Fatal(err) + } + } +} + +// committedSet is the set of paths HEAD changed. +func committedSet(t *testing.T, dir string) map[string]bool { + t.Helper() + set := map[string]bool{} + for _, f := range strings.Split(strings.TrimSpace(committedFiles(t, dir)), "\n") { + set[f] = true + } + return set +} + +// The never-commit folders are the project's own, beside the .mpr, where +// Studio Pro's .gitignore anchors them (/deployment/). Matched at any depth, a +// user folder of the same name was silently left out of the commit. +func TestGitCommit_NestedFolderNamedLikeJunkIsCommitted(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + writeFiles(t, dir, "resources/deployment/endpoints.xml", "javascriptsource/theme-cache/x.js", "deployment/run.log", "theme-cache/a.css") + + var out bytes.Buffer + if err := commitRun(mprPath, "nested", false, false, &out); err != nil { + t.Fatalf("commitRun: %v\n%s", err, out.String()) + } + files := committedSet(t, dir) + for _, keep := range []string{"resources/deployment/endpoints.xml", "javascriptsource/theme-cache/x.js"} { + if !files[keep] { + t.Errorf("%s is not the project's own folder and must be committed:\n%s", keep, out.String()) + } + } + for _, junk := range []string{"deployment/run.log", "theme-cache/a.css"} { + if files[junk] { + t.Errorf("%s is the project's own and must be left out", junk) + } + } +} + +// With the project in a subfolder of the repository, its folders are under +// that subfolder; the same names elsewhere in the repository are not its own. +func TestGitCommit_JunkAnchoredAtProjectInSubfolder(t *testing.T) { + root := t.TempDir() + doctorEnv(t, root) + gitIn(t, root, "init", "-q", "-b", "feature") + app := filepath.Join(root, "app") + if err := os.MkdirAll(app, 0o755); err != nil { + t.Fatal(err) + } + writeProject(t, app, "11.13.0", baseUnits(t)) + gitIn(t, root, "add", "-A") + gitIn(t, root, "commit", "-q", "-m", "base") + writeFiles(t, root, "app/deployment/run.log", "app/resources/deployment/endpoints.xml", "tools/deployment/deploy.sh") + + var out bytes.Buffer + if err := commitRun(filepath.Join(app, "App.mpr"), "sub", false, false, &out); err != nil { + t.Fatalf("commitRun: %v\n%s", err, out.String()) + } + files := committedSet(t, root) + if files["app/deployment/run.log"] { + t.Error("app/deployment/ is the project's own and must be left out") + } + for _, keep := range []string{"app/resources/deployment/endpoints.xml", "tools/deployment/deploy.sh"} { + if !files[keep] { + t.Errorf("%s must be committed:\n%s", keep, out.String()) + } + } +} + // An unresolved merge must refuse the commit: `git add -A` would otherwise // stage the conflict markers and commit them. func TestGitCommit_RefusedDuringUnresolvedMerge(t *testing.T) { diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 2da3867e1..8219bcf62 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -196,8 +196,10 @@ In order it: 1. repairs ContentsHash mismatches from the files on disk (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit); 2. stages everything, then unstages the files Studio Pro never commits: - `project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, - `deployment/`, `theme-cache/`, `.mxcli/`; + `project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's + own `.mendix-cache/`, `deployment/`, `theme-cache/` and `.mxcli/`, the + folders Studio Pro, mxbuild and mxcli create beside the `.mpr`. A folder of + the same name further down, such as `resources/deployment/`, is committed; 3. runs `git commit` with your message and your git identity; 4. computes the mx_metadata note from the new commit and attaches it, and prints the `git push origin refs/notes/mx_metadata` command — From 58e8346f8be08791fefd3d6c242ab709d2de5cce Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 22:58:01 +0000 Subject: [PATCH 19/44] fix(widgets): read switch arms and statement-head guards in editorConfig hide rules A pluggable widget's enumeration that its editorConfig hides must hold its default, or mx check reports CE0463. mxcli refused that for the one Timeline rule it read (`groupEvents: false`), but `widget describe timeline` showed "2 of 6 editor hide-rules recognized". So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed check and exec and failed mx check (ako/mxcli#1762, mendixlabs/mxcli#1668). Two shapes were unread: - terser folds the statements before a `switch` into its head, `switch(A ? hide : hide, B || hide, e.groupByKey)`, so the THEN branch's guard is bounded by the `switch(` paren, which parseGuard took for a conditional group. A `switch(`/`if(` head directly in a function body is now statement-level; a call's paren still is not. - a hide in a `case`/`default` arm had no reading. switchArmCondition now states the arm's values as eq/in, or ne/notin over the other cases for a default arm, following fall-through and grouped labels. Anything it cannot state exactly (a guarded hide inside the arm, a non-literal label, an arm whose exit is unclear, a switch under an `if`) yields no rule, as before. Timeline is 6 of 6 (3.1.1, 3.2.2, 3.2.3), so each hidden enumeration now gets MDL-WIDGET10's existing verdict: an error for a non-default value, which exec's preflight refuses before writing; a warning at the default; written at the default when left out. Across every installed package dumped before and after, only Timeline and HeatMap gained rules (HeatMap folds an `if(` head the same way). MDL-WIDGET10 names an in/notin set instead of "matches". WidgetDefGeneratorVersion goes to 20 so cached definitions regenerate. The custom-mode warning on title/description/timeIndication came from the hand-written fallback table, whose comment claimed only TextTemplates can cause CE0463. That comment, and the writer's, are corrected. File Uploader's ternary-wrapped hidePropertiesIn (ako/mxcli#258 item 1) was already lifted by the else-if chain support of mendixlabs/mxcli#1583. It is pinned by a test against File Uploader 2.5.0. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: the six hidden variants were CE0463 and the three controls 0 errors before the fix; check now refuses exactly those six. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...meline-hidden-enumeration-switch-arms.json | 1 + CHANGELOG.md | 1 + .../exec_preflight_timeline_1762_test.go | 103 ++++ docs-site/src/guides/pluggable-widgets.md | 13 + .../1762-timeline-hidden-enumeration.mdl | 114 ++++ mdl/backend/widgetobj/builder.go | 12 +- mdl/executor/editorconfig_extract.go | 47 +- mdl/executor/editorconfig_switch.go | 496 ++++++++++++++++++ mdl/executor/editorconfig_switch_1762_test.go | 235 +++++++++ .../validate_timeline_hidden_1762_test.go | 136 +++++ mdl/executor/validate_widgets.go | 19 + mdl/executor/widget_defs.go | 21 +- mdl/executor/widget_engine.go | 7 +- 13 files changed, 1186 insertions(+), 19 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json create mode 100644 cmd/mxcli/exec_preflight_timeline_1762_test.go create mode 100644 mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl create mode 100644 mdl/executor/editorconfig_switch.go create mode 100644 mdl/executor/editorconfig_switch_1762_test.go create mode 100644 mdl/executor/validate_timeline_hidden_1762_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json new file mode 100644 index 000000000..e8d3f6949 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-28","symptom":"A Timeline enumeration the widget hides, set to a non-default value — `customVisualization: true, groupByDayOptions: 'dayMonth'`, or `groupByKey: 'day', groupByMonthOptions: 'monthYear'` — passes `mxcli check` and `exec`, and mx check reports \"[CE0463] The definition of this widget has changed\" at Timeline 'tlD' / 'tlG'. `mxcli widget describe timeline` says \"2 of 6 editor hide-rules recognized\"; `groupEvents: false` plus a hidden value was already an MDL-WIDGET10 error (ako/mxcli#1762, mendixlabs/mxcli#1668)","cause":"Two editorConfig shapes went unread. terser folds the statements before a `switch` into its head — `switch(e.customVisualization ? hide([…]) : hide([…]), e.groupEvents || hide([…]), e.groupByKey)` — so the THEN branch's guard is bounded by the `switch(` paren, which parseGuard rejected as a conditional group (the else branch was lifted, through trailingExpr). And a hide in a `case`/`default` arm had no reading at all: its `case \"day\":` colon reached the ternary-else path and found no `?`. The template warnings the reporter saw in custom mode came from the hand-written fallback table in widget_defs.go, which listed only the TextTemplate keys of that same hide, on the stated premise that enumerations cannot cause CE0463","file":"mdl/executor/editorconfig_switch.go (switchArmCondition, switchArms, statementHeadParen, atFunctionStatementLevel), editorconfig_extract.go (the switch-arm branch before parseGuard; parseGuard's `(` boundary), validate_widgets.go (visibilityCondWord in/notin), widget_engine.go (WidgetDefGeneratorVersion 20)","insight":"**A known rule elsewhere is not the extractor knowing it.** The custom-mode warning made rule (1) look recognised, and it was the reason to ask why the enumerations in the same list were missed: `widget describe` (extractor only) showed no `title` rule, while the generated .def.json had one, merged from the hand table. Compare the two sources first when a rule half-works. **The reporter's framing (`the true branch of a ternary`) was not the cause**: the existing TimelinePattern test already lifted a statement-level THEN branch, and only the minified file showed that terser had moved it into the `switch(` head. Read the shipped JS, not the de-minified source. **Measure the matrix, then make it the oracle**: one widget per variant through `mx check --no-update-widgets` on 11.13.0 gave 6 CE0463 and 3 clean, and the extractor test, the check test and the bug-test all encode those nine. **Policy follows the existing split, not a second one**: hidden + non-default is refused (MDL-WIDGET10 error, exec preflight) and hidden + left out is written at the default, as rule (3) and the column-SET reset (mendixlabs/mxcli#1519) already do. **Bound the blast radius by dumping lifted rules for every installed .mpk before and after**: only Timeline and HeatMap changed. HeatMap folds into an `if(` head the same way, and its new rule is correct. **The scratchpad can be shared between agents**: work in a private subdirectory","refs":["ako/mxcli#1762","mendixlabs/mxcli#1668","ako/mxcli#258","mendixlabs/mxcli#1583"],"ce":["CE0463"],"rules":["MDL-WIDGET10"],"test":"mdl/executor/editorconfig_switch_1762_test.go; validate_timeline_hidden_1762_test.go; cmd/mxcli/exec_preflight_timeline_1762_test.go; mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl"} diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ff88359a..c33507cc8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/cmd/mxcli/exec_preflight_timeline_1762_test.go b/cmd/mxcli/exec_preflight_timeline_1762_test.go new file mode 100644 index 000000000..e948758ed --- /dev/null +++ b/cmd/mxcli/exec_preflight_timeline_1762_test.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/mdl/deprecation" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// ako/mxcli#1762 (mendixlabs/mxcli#1668), the reporter's script. Reported: check +// and exec both pass, and mx check says "[CE0463] The definition of this widget +// has changed" at Timeline 'tlD' (groupByDayOptions, hidden in custom mode) and +// 'tlG' (groupByMonthOptions, hidden when groupByKey is 'day'). exec must refuse +// it before writing anything, naming both — as it already did for the one +// Timeline rule it knew, `groupEvents: false`. +const timeline1762Setup = `create module ModTL; +create non-persistent entity ModTL.Ev ( Line: String(200), WhoText: String(200), WhenText: String(50), At: DateTime ); +create microflow ModTL.DS_Events () returns List of ModTL.Ev as $Events +begin + $Events = create list of ModTL.Ev; + $E = create ModTL.Ev ( Line = 'Created', WhoText = 'someone', WhenText = '10:00', At = [%CurrentDateTime%] ); + add $E to $Events; + return $Events; +end; +` + +func timeline1762Pages(dayOptions, monthOptions string) string { + return `create page ModTL.PageD ( Title: 'D', Layout: Atlas_Core.Atlas_Default ) { + timeline tlD ( + data: MICROFLOW ModTL.DS_Events, + customVisualization: true, groupEvents: true, groupAttribute: At, + groupByKey: 'day', ` + dayOptions + `ungroupedEventsPosition: 'end' + ) { + customtitle d2 { dynamictext d2t (Content: '{1}', ContentParams: ({1} = Line)) } + } +}; +create page ModTL.PageG ( Title: 'G', Layout: Atlas_Core.Atlas_Default ) { + timeline tlG ( + data: MICROFLOW ModTL.DS_Events, + title: '{1}', titleParams: ({1} = $currentObject/Line), + customVisualization: false, groupEvents: true, groupAttribute: At, + groupByKey: 'day', ` + monthOptions + `ungroupedEventsPosition: 'end' + ) +}; +` +} + +func TestExecPreflightRefusesHiddenTimelineEnumerations(t *testing.T) { + src := filepath.Join("..", "..", "testdata", "expr-checker") + if _, err := os.Stat(filepath.Join(src, "minimal.mpr")); err != nil { + t.Skipf("expr-checker fixture not found: %v", err) + } + dir := t.TempDir() + if err := os.CopyFS(dir, os.DirFS(src)); err != nil { + t.Fatal(err) + } + mpr := filepath.Join(dir, "minimal.mpr") + exec := executor.New(io.Discard) + exec.SetBackendFactory(func() backend.FullBackend { return modelsdkbackend.New() }) + if err := exec.Execute(&ast.ConnectStmt{Path: mpr}); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = exec.Execute(&ast.DisconnectStmt{}) }) + + preflight := func(script string) (string, string) { + t.Helper() + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatal(errs[0]) + } + var out bytes.Buffer + refusal := execPreflight(exec, prog, mpr, "s.mdl", false, false, false, deprecation.Warn, &out, false) + return refusal, out.String() + } + + refusal, out := preflight(timeline1762Setup + + timeline1762Pages("groupByDayOptions: 'dayMonth', ", "groupByMonthOptions: 'monthYear', ")) + if refusal == "" || !strings.Contains(refusal, "Nothing was written") { + t.Fatalf("want the pre-flight to refuse the script, got %q\n%s", refusal, out) + } + for _, want := range []string{"`tlD`", "`groupByDayOptions`", "`tlG`", "`groupByMonthOptions`"} { + if !strings.Contains(out, want) { + t.Errorf("pre-flight output must name %s:\n%s", want, out) + } + } + + // CONTROL: the same pages without the hidden values (variant b of the + // report, which mx check passes) run. + if refusal, out := preflight(timeline1762Setup + timeline1762Pages("", "")); refusal != "" { + t.Errorf("the pages without hidden values were refused: %s\n%s", refusal, out) + } +} diff --git a/docs-site/src/guides/pluggable-widgets.md b/docs-site/src/guides/pluggable-widgets.md index a83a59eb0..6ef086397 100644 --- a/docs-site/src/guides/pluggable-widgets.md +++ b/docs-site/src/guides/pluggable-widgets.md @@ -51,6 +51,19 @@ version you actually have at the moment you create the widget. rules** and applies them, so a freshly-created widget carries exactly the defaults Studio Pro would give it. + A property the widget hides must hold its default, or the build fails with CE0463. + So a hidden property you leave out is written at its default; one you set to its + default draws an MDL-WIDGET10 warning (the value is ignored); and one you set to + anything else is an MDL-WIDGET10 **error**, which `exec` refuses before writing. + For example, a + Timeline grouped by `'day'` hides `groupByMonthOptions`, and one with + `customVisualization: true` hides both format enumerations. + + This only covers rules mxcli recognises. `mxcli widget describe -p app.mpr` + ends with "*N* of *M* editor hide-rules recognized". A rule it cannot read hides + nothing as far as `check` knows, so a value set behind one reaches `mx check` + unwarned. + The result: widgets created by mxcli open cleanly across Mendix 10.x and 11.x with the widget packages bundled with those releases — verified on Mendix 11.12 with Data Widgets 3.4: a project authored entirely by mxcli reports **0** CE0463. diff --git a/mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl b/mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl new file mode 100644 index 000000000..3cbb491b3 --- /dev/null +++ b/mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl @@ -0,0 +1,114 @@ +mdl 1; +-- ============================================================================ +-- Issue ako/mxcli#1762 (mendixlabs/mxcli#1668) — a Timeline enumeration the +-- widget hides, set to a non-default value, fails the build with CE0463 +-- ============================================================================ +-- +-- The Timeline's editorConfig hides properties six ways: +-- +-- e.customVisualization ? hide(title, description, icon, timeIndication, +-- groupByDayOptions, groupByMonthOptions) (1) +-- : hide(customIcon, customTitle, …) (2) +-- e.groupEvents || hide(groupAttribute, groupByKey, groupByDayOptions, …) (3) +-- switch (e.groupByKey) { case "day": hide(groupByMonthOptions) (4) +-- case "month": hide(groupByDayOptions) (5) +-- default: hide(groupByDayOptions, +-- groupByMonthOptions) } (6) +-- +-- mxcli read (2) and (3) only. Measured on mxbuild 11.13.0 with Timeline 3.2.2, +-- one widget per variant, nothing else hidden set: +-- +-- customVisualization: true, groupByDayOptions: 'dayMonth' → CE0463 (1) +-- customVisualization: true, groupByKey: 'month', +-- groupByMonthOptions: 'monthYear' → CE0463 (1) +-- groupByKey: 'day', groupByMonthOptions: 'monthYear' → CE0463 (4) +-- groupByKey: 'month', groupByDayOptions: 'fullDate' → CE0463 (5) +-- groupByKey: 'year', groupByDayOptions / groupByMonthOptions set → CE0463 (6) +-- the same values where they are visible, or left out → 0 errors +-- +-- `check` and `exec` passed all of them. Now each is refused with MDL-WIDGET10 +-- — the verdict (3) already got — naming the property, the rule and the +-- default; one the script leaves out is written at that default. +-- +-- Verify (needs -p: the rules are lifted from the project's installed .mpk): +-- +-- mxcli check 1762-timeline-hidden-enumeration.mdl -p app.mpr --references +-- → passes; uncomment a statement below to see the MDL-WIDGET10 error +-- mxcli exec 1762-timeline-hidden-enumeration.mdl -p app.mpr +-- mxcli docker check -p app.mpr --no-update-widgets # 0 errors +-- describe page M1762.PageD, then exec it # unchanged +-- ============================================================================ + +create module M1762; + +create non-persistent entity M1762.Ev ( + Line: String(200), + WhoText: String(200), + At: DateTime +); + +create microflow M1762.DS_Events () returns List of M1762.Ev as $Events +begin + $Events = create list of M1762.Ev; + $E = create M1762.Ev ( Line = 'Created', WhoText = 'someone', At = [%CurrentDateTime%] ); + add $E to $Events; + return $Events; +end; + +-- The reporter's tlD, corrected: custom mode hides the format enumerations, so +-- they are left out (written at their defaults). +create or modify page M1762.PageD ( Title: 'D', Layout: Atlas_Core.Atlas_Default ) { + timeline tlD ( + data: MICROFLOW M1762.DS_Events, + customVisualization: true, groupEvents: true, groupAttribute: At, + groupByKey: 'day', ungroupedEventsPosition: 'end' + ) { + customtitle d2 { dynamictext d2t (Content: '{1}', ContentParams: ({1} = Line)) } + } +}; + +-- The reporter's tlG, corrected: grouping by day shows only groupByDayOptions. +create or modify page M1762.PageG ( Title: 'G', Layout: Atlas_Core.Atlas_Default ) { + timeline tlG ( + data: MICROFLOW M1762.DS_Events, + title: '{1}', titleParams: ({1} = $currentObject/Line), + customVisualization: false, groupEvents: true, groupAttribute: At, + groupByKey: 'day', groupByDayOptions: 'fullDate', ungroupedEventsPosition: 'end' + ) +}; + +-- Grouping by month shows only groupByMonthOptions. +create or modify page M1762.PageM ( Title: 'M', Layout: Atlas_Core.Atlas_Default ) { + timeline tlM ( + data: MICROFLOW M1762.DS_Events, + title: '{1}', titleParams: ({1} = $currentObject/Line), + groupAttribute: At, groupByKey: 'month', groupByMonthOptions: 'monthYear' + ) +}; + +-- The failing shapes, kept as documentation. Uncomment one and re-run +-- `mxcli check ... -p app.mpr`: MDL-WIDGET10 is an error, and `mx check` +-- reports CE0463 on the same widget. +-- +-- (1) hidden in custom mode: +-- +-- create or modify page M1762.PageD1 ( Title: 'x', Layout: Atlas_Core.Atlas_Default ) { +-- timeline tlD1 ( data: MICROFLOW M1762.DS_Events, customVisualization: true, +-- groupAttribute: At, groupByKey: 'day', groupByDayOptions: 'dayMonth' ) { +-- customtitle d3 { dynamictext d3t (Content: '{1}', ContentParams: ({1} = Line)) } +-- } +-- }; +-- +-- (4) grouping by day hides the month format: +-- +-- create or modify page M1762.PageG1 ( Title: 'x', Layout: Atlas_Core.Atlas_Default ) { +-- timeline tlG1 ( data: MICROFLOW M1762.DS_Events, title: '{1}', titleParams: ({1} = $currentObject/Line), +-- groupAttribute: At, groupByKey: 'day', groupByMonthOptions: 'monthYear' ) +-- }; +-- +-- (6) grouping by year hides both: +-- +-- create or modify page M1762.PageY1 ( Title: 'x', Layout: Atlas_Core.Atlas_Default ) { +-- timeline tlY1 ( data: MICROFLOW M1762.DS_Events, title: '{1}', titleParams: ({1} = $currentObject/Line), +-- groupAttribute: At, groupByKey: 'year', groupByDayOptions: 'dayMonth' ) +-- }; diff --git a/mdl/backend/widgetobj/builder.go b/mdl/backend/widgetobj/builder.go index f89bf4e2d..8f1d0408c 100644 --- a/mdl/backend/widgetobj/builder.go +++ b/mdl/backend/widgetobj/builder.go @@ -882,9 +882,11 @@ func (ob *Builder) EnsureRequiredObjectLists() {} // ApplyPropertyVisibility nulls the TextTemplate of any TextTemplate-typed // property the rules mark as hidden under the widget's current configuration. -// Non-TextTemplate properties are left untouched: only the populated-vs-null -// ClientTemplate choice triggers CE0463, so Phase 1 scopes the action to -// TextTemplate. The widget's current primitive values (read from the assembled +// Non-TextTemplate properties are left untouched HERE, not because they are +// harmless: a hidden enumeration at a non-default value is CE0463 too +// (ako/mxcli#1762). The engine writes one the script does not name at its +// declared default (hiddenUnnamedProperties), and MDL-WIDGET10 refuses one it +// does. The widget's current primitive values (read from the assembled // object) drive rule evaluation, so a rule keyed on e.g. `type` sees the value // just set from MDL. @@ -897,8 +899,8 @@ func (ob *Builder) ApplyPropertyVisibility(rules []types.WidgetVisibilityRule) { // the updated object. It is the engine-agnostic form of ApplyPropertyVisibility // so build paths that construct a widget object without a full Builder (e.g. the // filter-widget build path) can apply the same #574 nulling. Non-TextTemplate -// properties are left untouched — only the populated-vs-null ClientTemplate -// choice triggers CE0463. +// properties are left untouched here; see ApplyPropertyVisibility for where a +// hidden enumeration's default is enforced instead. func ApplyVisibilityRules(object bson.D, propertyTypeIDs map[string]pages.PropertyTypeIDEntry, rules []types.WidgetVisibilityRule) bson.D { if len(rules) == 0 { return object diff --git a/mdl/executor/editorconfig_extract.go b/mdl/executor/editorconfig_extract.go index 6ced44279..2450a8b72 100644 --- a/mdl/executor/editorconfig_extract.go +++ b/mdl/executor/editorconfig_extract.go @@ -127,8 +127,9 @@ func extractVisibilityRulesFromMPK(mpkPath, widgetID string) []types.WidgetVisib // widget's compiled editorConfig.js into declarative WidgetVisibilityRules. // // It recognizes the dominant `getProperties` idioms — `"V"===ref && hide(...)`, -// `"V"!==ref && hide(...)`, `ref && hide(...)`, `ref || hide(...)`, and -// `ref ? hide(...) : …` — where `ref` is `obj.prop` or a locally-aliased +// `"V"!==ref && hide(...)`, `ref && hide(...)`, `ref || hide(...)`, +// `ref ? hide(...) : …`, and a hide in a `switch(ref){case "V": …}` arm — where +// `ref` is `obj.prop` or a locally-aliased // identifier resolved within the enclosing function scope. Everything it cannot // lift (object-list-nested hides, compound/computed guards, unresolved aliases) // is counted in the returned stats so callers can gauge coverage; unrecognized @@ -159,7 +160,21 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit if listKey != "" { itemIdent = enclosingForEachParam(js, callStart) } - cond, guardText, ok, conjunctive, chain := parseGuard(js, callStart, itemIdent) + // A hide in a `switch` arm runs on the arm's case values, not on a guard + // beside it, so it is read from the switch and never from parseGuard — + // whose `:` handling would take the arm's label for a ternary's else. + var ( + cond types.WidgetVisibilityCondition + guardText string + conjunctive bool + chain []types.WidgetVisibilityCondition + ) + armCond, inSwitchArm, armOK := switchArmCondition(js, callStart, itemIdent) + if inSwitchArm { + cond, ok = armCond, armOK + } else { + cond, guardText, ok, conjunctive, chain = parseGuard(js, callStart, itemIdent) + } // A guard inside `outer ? ( … inner && hide(x) … )` states only the INNER // term; the branch runs on outer too. Collect the enclosing group guards // so the rule carries the whole conjunction — Combo box nests three deep, @@ -254,13 +269,19 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit // guard alone would mark both branches hidden together (#956). if len(condKeys) > 0 { aliases := enclosingAliases(js, callStart) + // In a switch arm the element's condition holds only where the arm + // runs, so the arm's condition is a term of its own. + condExtra := extra + if inSwitchArm { + condExtra = append([]types.WidgetVisibilityCondition{cond}, extra...) + } for _, tk := range condKeys { c, ok := guardToCondition(tk.guard, tk.falsy, aliases, itemIdent) if !ok { stats.SkippedComplex++ continue } - sig := listKey + "\x00" + tk.key + "\x00" + c.PropertyKey + c.Operator + c.Value + c.Scope + condsSig(extra) + sig := listKey + "\x00" + tk.key + "\x00" + c.PropertyKey + c.Operator + c.Value + c.Scope + condsSig(condExtra) if seen[sig] { continue } @@ -270,7 +291,7 @@ func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, edit PropertyKey: tk.key, ListPropertyKey: listKey, HiddenWhen: &cc, - And: extra, + And: condExtra, }) } } @@ -810,6 +831,22 @@ func parseGuard(js string, callStart int, itemIdent string) (types.WidgetVisibil if !falsy { return types.WidgetVisibilityCondition{}, guard, false, false, nil } + case '(': + // The paren heads a STATEMENT rather than grouping a guarded branch: + // terser folds the expression statements before a `switch`/`if` into + // its head, and their contents run unconditionally. Timeline's first + // hide is the THEN branch of + // + // switch(e.customVisualization ? hide([…]) : hide([…]), e.groupEvents || hide([…]), e.groupByKey){…} + // + // and was dropped as "one operand of a compound condition" while its + // else branch, read through trailingExpr, was lifted (ako/mxcli#1762). + // A call's paren is not this case — the call can itself be guarded. + // boundaryAt indexes js only while pre is still a prefix of it, which + // an else-branch guard (cut out by ternaryCondition) is not. + if elseBranch || !strings.HasPrefix(js, pre) || !statementHeadParen(js, boundaryAt) { + return types.WidgetVisibilityCondition{}, guard, false, false, nil + } default: return types.WidgetVisibilityCondition{}, guard, false, false, nil } diff --git a/mdl/executor/editorconfig_switch.go b/mdl/executor/editorconfig_switch.go new file mode 100644 index 000000000..056bff4dd --- /dev/null +++ b/mdl/executor/editorconfig_switch.go @@ -0,0 +1,496 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// A widget's getProperties can choose what to hide with a `switch` on one of its +// enumerations. Timeline 3.2 does, and three of its six hide rules were unread +// (ako/mxcli#1762, mendixlabs/mxcli#1668): +// +// switch (…, e.groupByKey) { +// case "day": hidePropertyIn(t, e, "groupByMonthOptions"); break; +// case "month": hidePropertyIn(t, e, "groupByDayOptions"); break; +// default: hidePropertiesIn(t, e, ["groupByDayOptions", "groupByMonthOptions"]); +// } +// +// A hide in an arm fires on the values that enter the arm: its own labels, the +// labels of every arm that falls through into it, and — when a default arm is +// among those — every value no case names. That is one `eq`/`in` condition, or +// one `ne`/`notin` over the cases the switch does name. +// +// Exact or nothing, like the rest of the extractor: a hide carrying its own +// guard inside the arm, a label that is not a string literal, a switch that is +// not a statement of a function body, or a discriminant that is not one of the +// widget's properties yields no rule — which reads as "not hidden", the state +// every one of these was in before. + +// switchArm is one `case`/`default` arm of a switch body. +type switchArm struct { + labels []string // the case values; empty for a lone default + isDefault bool + start, end int // the arm's statements, as offsets into the switch body + exit armExit // whether control falls into the next arm +} + +// armExit is how control leaves a switch arm. +type armExit int + +const ( + armFallsThrough armExit = iota // runs on into the next arm + armTerminates // ends in break/return/throw/continue + armUnknown // cannot be told from its text (a block, a conditional break) +) + +// switchArmCondition reports whether the hide at callStart sits in a switch arm +// (inArm) and, when it does, the condition under which that arm runs. ok is +// false for an arm whose condition cannot be stated exactly; the caller then +// emits no rule and does NOT fall back to the guard parser, whose reading of +// the arm's `case "v":` label as a ternary's else could only be wrong. +func switchArmCondition(js string, callStart int, itemIdent string) (cond types.WidgetVisibilityCondition, inArm, ok bool) { + none := types.WidgetVisibilityCondition{} + open := enclosingOpenerQuoted(js, callStart) + if open < 0 || js[open] != '{' { + return none, false, false + } + head := strings.TrimRight(js[:open], " ") + if !strings.HasSuffix(head, ")") { + return none, false, false + } + parenOpen := matchingOpenParen(head) + if parenOpen < 0 { + return none, false, false + } + kwHead := strings.TrimRight(head[:parenOpen], " ") + if !hasKeywordSuffix(kwHead, "switch") { + return none, false, false + } + + // From here on the hide is in a switch arm. + kwStart := len(kwHead) - len("switch") + if !atFunctionStatementLevel(js, kwStart) { + return none, true, false // the switch runs only under some outer condition + } + // terser folds the preceding statements into the head — `switch(A, B, e.k)` — + // and the switch is on the LAST of them. + parts := splitTopLevelCommas(head[parenOpen+1 : len(head)-1]) + m := refRE.FindStringSubmatch(strings.TrimSpace(parts[len(parts)-1])) + if m == nil || m[1] == "!" { + return none, true, false + } + key, scope, resolved := resolveRef(m[2], enclosingAliases(js, kwStart), itemIdent) + if !resolved { + return none, true, false // the platform argument, or a computed local + } + body, balanced := balancedArgs(js, open+1) + if !balanced { + return none, true, false + } + arms, parsed := switchArms(body) + if !parsed { + return none, true, false + } + at := callStart - (open + 1) + idx := -1 + for i, a := range arms { + if at >= a.start && at < a.end { + idx = i + } + } + if idx < 0 || !onlyHideStatementsBefore(body[arms[idx].start:at]) { + return none, true, false + } + + first := idx +walk: + for first > 0 { + switch arms[first-1].exit { + case armTerminates: + break walk + case armUnknown: + // Treating it either way could be wrong: as falling through, this + // rule claims its values; as terminating, it misses them. + return none, true, false + } + first-- // that arm falls through into this one + } + var entered []string + hasDefault := false + for _, a := range arms[first : idx+1] { + entered = append(entered, a.labels...) + hasDefault = hasDefault || a.isDefault + } + op, vals := "in", uniqueStrings(entered) + if hasDefault { + // Entered by every value no case names, plus this run's own cases: the + // complement of the OTHER arms' cases. + var all []string + for _, a := range arms { + all = append(all, a.labels...) + } + op, vals = "notin", withoutStrings(uniqueStrings(all), vals) + } + switch { + case len(vals) == 0: + return none, true, false // every value enters: unconditional, not a condition + case len(vals) == 1 && op == "in": + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: "eq", Value: vals[0], Scope: scope}, true, true + case len(vals) == 1: + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: "ne", Value: vals[0], Scope: scope}, true, true + } + for _, v := range vals { + if strings.ContainsRune(v, ',') { + return none, true, false // the joined set would decode wrong + } + } + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: op, Value: strings.Join(vals, ","), Scope: scope}, true, true +} + +// switchArms splits a switch body into its arms. ok is false for anything but +// `case "literal":` and `default:` labels, or for text before the first label. +func switchArms(body string) ([]switchArm, bool) { + var arms []switchArm + depth := 0 + inStr := byte(0) + for i := 0; i < len(body); i++ { + c := body[i] + if inStr != 0 { + if c == '\\' { + i++ + } else if c == inStr { + inStr = 0 + } + continue + } + switch c { + case '"', '\'', '`': + inStr = c + continue + case '(', '[', '{': + depth++ + continue + case ')', ']', '}': + depth-- + continue + } + if depth != 0 || (i > 0 && (isIdentByte(body[i-1]) || body[i-1] == '.')) { + continue + } + var arm switchArm + var next int + switch { + case hasKeywordAt(body, i, "case"): + label, end, ok := switchCaseLabel(body, i+len("case")) + if !ok { + return nil, false + } + arm.labels, next = []string{label}, end + case hasKeywordAt(body, i, "default"): + j := skipSpaces(body, i+len("default")) + if j >= len(body) || body[j] != ':' { + return nil, false + } + arm.isDefault, next = true, j+1 + default: + continue + } + if len(arms) == 0 { + if strings.TrimSpace(body[:i]) != "" { + return nil, false + } + } else { + arms[len(arms)-1].end = i + } + arm.start, arm.end = next, len(body) + arms = append(arms, arm) + i = next - 1 + } + if len(arms) == 0 { + return nil, false + } + for i := range arms { + arms[i].exit = exitOf(body[arms[i].start:arms[i].end]) + } + return arms, true +} + +// switchCaseLabel reads the `"value":` after a `case` keyword, returning the value and +// the offset just past the colon. +func switchCaseLabel(body string, from int) (string, int, bool) { + j := skipSpaces(body, from) + if j >= len(body) || (body[j] != '"' && body[j] != '\'') { + return "", 0, false + } + q := body[j] + k := strings.IndexByte(body[j+1:], q) + if k < 0 { + return "", 0, false + } + value := body[j+1 : j+1+k] + if strings.ContainsRune(value, '\\') { + return "", 0, false + } + end := skipSpaces(body, j+1+k+1) + if end >= len(body) || body[end] != ':' { + return "", 0, false + } + return value, end + 1, true +} + +// abruptKeywords leave a switch arm. +var abruptKeywords = []string{"break", "return", "throw", "continue"} + +// exitOf reports how control leaves an arm. Only two readings are certain: the +// last statement is itself a break/return/throw/continue, or the arm holds none +// of them and no block that could hide one. Anything else — `{ …; break }`, +// `if (c) break; …` — is unknown. +func exitOf(arm string) armExit { + stmts := splitTopLevel(arm, ';') + for i := len(stmts) - 1; i >= 0; i-- { + s := strings.TrimSpace(stmts[i]) + if s == "" { + continue + } + for _, kw := range abruptKeywords { + if hasKeywordAt(s, 0, kw) { + return armTerminates + } + } + if mayLeaveEarly(arm) { + return armUnknown + } + return armFallsThrough + } + return armFallsThrough // an empty arm (a grouped label) falls through +} + +// mayLeaveEarly reports whether s, outside its string literals, holds a block or +// an abrupt keyword anywhere. +func mayLeaveEarly(s string) bool { + inStr := byte(0) + for i := 0; i < len(s); i++ { + c := s[i] + if inStr != 0 { + if c == '\\' { + i++ + } else if c == inStr { + inStr = 0 + } + continue + } + switch c { + case '"', '\'', '`': + inStr = c + continue + case '{': + return true + } + if i > 0 && (isIdentByte(s[i-1]) || s[i-1] == '.') { + continue + } + for _, kw := range abruptKeywords { + if hasKeywordAt(s, i, kw) { + return true + } + } + } + return false +} + +// onlyHideStatementsBefore reports whether the text of an arm before a hide is +// nothing but earlier hide calls — `case "v": hide(a), hide(b); hide(c)` — so +// the hide is guarded by the arm alone and not by a condition of its own. +func onlyHideStatementsBefore(prefix string) bool { + p := trimNamespacePrefix(strings.TrimRight(prefix, " ")) + for { + if strings.TrimSpace(p) == "" { + return true + } + if !strings.HasSuffix(p, ";") && !strings.HasSuffix(p, ",") { + return false + } + call := strings.TrimRight(p[:len(p)-1], " ") + if !strings.HasSuffix(call, ")") { + return false + } + open := matchingOpenParen(call) + if open < 0 { + return false + } + name := strings.TrimRight(call[:open], " ") + loc := hideCallTailRE.FindStringIndex(name) + if loc == nil || loc[1] != len(name) { + return false + } + p = trimNamespacePrefix(strings.TrimRight(name[:loc[0]], " ")) + } +} + +func trimNamespacePrefix(p string) string { + if loc := nsPrefixRE.FindStringIndex(p); loc != nil { + return strings.TrimRight(p[:loc[0]], " ") + } + return p +} + +// statementHeadParen reports whether the '(' at open is the head of a `switch` +// or `if` statement sitting directly in a function body. Its contents then run +// whenever the function does, so a guard bounded by it is statement-level. +func statementHeadParen(js string, open int) bool { + head := strings.TrimRight(js[:open], " ") + for _, kw := range []string{"switch", "if"} { + if hasKeywordSuffix(head, kw) { + return atFunctionStatementLevel(js, len(head)-len(kw)) + } + } + return false +} + +// atFunctionStatementLevel reports whether the statement starting at pos is one +// of a function body's own statements — not an `else`'s, not under an `if` +// without braces, not inside a nested block. Anything else runs under a +// condition the rule would not carry. +func atFunctionStatementLevel(js string, pos int) bool { + before := strings.TrimRight(js[:pos], " ") + if before != "" { + switch before[len(before)-1] { + case '{', ';', '}': + default: + return false + } + } + open := enclosingOpenerQuoted(js, pos) + return open >= 0 && js[open] == '{' && isFunctionBody(js, open) +} + +// isFunctionBody reports whether the '{' at open opens a function's body: +// `function(…){`, `function name(…){`, or an arrow's `=>{`. +func isFunctionBody(js string, open int) bool { + head := strings.TrimRight(js[:open], " ") + if strings.HasSuffix(head, "=>") { + return true + } + if !strings.HasSuffix(head, ")") { + return false + } + params := matchingOpenParen(head) + if params < 0 { + return false + } + name := strings.TrimRight(head[:params], " ") + if hasKeywordSuffix(name, "function") { + return true + } + i := len(name) + for i > 0 && isIdentByte(name[i-1]) { + i-- + } + return i < len(name) && hasKeywordSuffix(strings.TrimRight(name[:i], " "), "function") +} + +// enclosingOpenerQuoted is enclosingOpener skipping quoted strings, so a caption +// holding a bracket does not throw the count off. +func enclosingOpenerQuoted(js string, pos int) int { + depth := 0 + inStr := byte(0) + for i := pos - 1; i >= 0; i-- { + c := js[i] + if inStr != 0 { + if c == inStr && (i == 0 || js[i-1] != '\\') { + inStr = 0 + } + continue + } + switch c { + case '"', '\'': + inStr = c + case '}', ')', ']': + depth++ + case '{', '(', '[': + if depth == 0 { + return i + } + depth-- + } + } + return -1 +} + +// hasKeywordSuffix reports whether s ends with the keyword kw as a whole word. +func hasKeywordSuffix(s, kw string) bool { + if !strings.HasSuffix(s, kw) { + return false + } + i := len(s) - len(kw) + return i == 0 || (!isIdentByte(s[i-1]) && s[i-1] != '.') +} + +// hasKeywordAt reports whether the keyword kw starts at s[i] as a whole word. +func hasKeywordAt(s string, i int, kw string) bool { + if !strings.HasPrefix(s[i:], kw) { + return false + } + end := i + len(kw) + return end == len(s) || !isIdentByte(s[end]) +} + +func skipSpaces(s string, i int) int { + for i < len(s) && s[i] == ' ' { + i++ + } + return i +} + +// splitTopLevel splits s on sep where it is not nested in a bracket or string. +func splitTopLevel(s string, sep byte) []string { + var parts []string + depth := 0 + inStr := byte(0) + start := 0 + for i := 0; i < len(s); i++ { + c := s[i] + if inStr != 0 { + if c == '\\' { + i++ + } else if c == inStr { + inStr = 0 + } + continue + } + switch c { + case '"', '\'', '`': + inStr = c + case '(', '[', '{': + depth++ + case ')', ']', '}': + depth-- + case sep: + if depth == 0 { + parts = append(parts, s[start:i]) + start = i + 1 + } + } + } + return append(parts, s[start:]) +} + +func withoutStrings(ss, drop []string) []string { + out := ss[:0:0] + for _, s := range ss { + keep := true + for _, d := range drop { + if s == d { + keep = false + break + } + } + if keep { + out = append(out, s) + } + } + return out +} diff --git a/mdl/executor/editorconfig_switch_1762_test.go b/mdl/executor/editorconfig_switch_1762_test.go new file mode 100644 index 000000000..6864c6c3c --- /dev/null +++ b/mdl/executor/editorconfig_switch_1762_test.go @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: Apache-2.0 + +// The Timeline's hide rules (ako/mxcli#1762, mendixlabs/mxcli#1668). +// +// A pluggable widget's enumeration that its editorConfig hides must be stored at +// its XML default, or mx check reports CE0463. mxcli refuses that for the one +// Timeline rule it recognised (`groupEvents: false`), and passed the rest: +// `mxcli widget describe timeline` reported "2 of 6 editor hide-rules +// recognized". Measured on mxbuild 11.13.0 with Timeline 3.2.2, one widget per +// variant, nothing else hidden on the page: +// +// customVisualization: true, groupByDayOptions: 'dayMonth' → CE0463 (1) +// customVisualization: true, groupByKey: 'month', groupByMonthOptions: 'monthYear' → CE0463 (1) +// groupByKey: 'day', groupByMonthOptions: 'monthYear' → CE0463 (4) +// groupByKey: 'month', groupByDayOptions: 'fullDate' → CE0463 (5) +// groupByKey: 'year', groupByDayOptions: 'dayMonth' → CE0463 (6) +// groupByKey: 'year', groupByMonthOptions: 'monthYear' → CE0463 (6) +// controls — custom mode with neither set; 'day' + groupByDayOptions; +// 'month' + groupByMonthOptions → 0 errors +// +// That matrix is this file's oracle. Two shapes went unread. terser folds the +// statements before the `switch` into its head, `switch(A ? hide : hide, B || +// hide, e.groupByKey)`, so rule (1)'s guard is bounded by the `switch(` paren, +// which the guard parser took for a conditional group. And the `case`/`default` +// arms of a switch had no reading at all. +package executor + +import ( + "slices" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// timelineGetProperties is Timeline 3.2.3's getProperties, verbatim from the +// marketplace package (3.2.2 differs only in the namespace identifier). +const timelineGetProperties = `exports.getProperties=function(e,t){switch(e.customVisualization?I.hidePropertiesIn(t,e,["title","description","icon","timeIndication","groupByDayOptions","groupByMonthOptions"]):I.hidePropertiesIn(t,e,["customIcon","customTitle","customEventDateTime","customDescription","customGroupHeader"]),e.groupEvents||I.hidePropertiesIn(t,e,["groupAttribute","groupByKey","groupByDayOptions","groupByMonthOptions","ungroupedEventsPosition","customGroupHeader"]),e.groupByKey){case"day":I.hidePropertyIn(t,e,"groupByMonthOptions");break;case"month":I.hidePropertyIn(t,e,"groupByDayOptions");break;default:I.hidePropertiesIn(t,e,["groupByDayOptions","groupByMonthOptions"])}return t};` + +// timelineCase is one configuration and whether prop is hidden under it. +type timelineCase struct { + prop string + cv string // customVisualization + ge string // groupEvents + key string // groupByKey + hidden bool +} + +var timelineCases = []timelineCase{ + // Rule (1), the ternary's THEN branch inside the switch head. + {"groupByDayOptions", "true", "true", "day", true}, // measured: CE0463 + {"groupByMonthOptions", "true", "true", "month", true}, // measured: CE0463 + {"title", "true", "true", "day", true}, + {"description", "true", "true", "day", true}, + {"timeIndication", "true", "true", "day", true}, + {"icon", "true", "true", "day", true}, + {"title", "false", "true", "day", false}, + {"icon", "false", "true", "day", false}, + // Rule (2), its ELSE branch — recognised before; the control. + {"customTitle", "false", "true", "day", true}, + {"customTitle", "true", "true", "day", false}, + // Rule (3) — recognised before; the control. + {"ungroupedEventsPosition", "false", "false", "day", true}, + {"ungroupedEventsPosition", "false", "true", "day", false}, + // Rules (4)–(6), the switch's arms. + {"groupByMonthOptions", "false", "true", "day", true}, // (4) measured: CE0463 + {"groupByDayOptions", "false", "true", "month", true}, // (5) measured: CE0463 + {"groupByDayOptions", "false", "true", "year", true}, // (6) measured: CE0463 + {"groupByMonthOptions", "false", "true", "year", true}, // (6) measured: CE0463 + {"groupByDayOptions", "false", "true", "day", false}, // measured: clean + {"groupByMonthOptions", "false", "true", "month", false}, // measured: clean +} + +func checkTimelineCases(t *testing.T, rules []types.WidgetVisibilityRule) { + t.Helper() + for _, c := range timelineCases { + values := map[string]string{"customVisualization": c.cv, "groupEvents": c.ge, "groupByKey": c.key} + hidden, found := ruleHidesUnder(rules, c.prop, values) + if !found && c.hidden { + t.Errorf("%s: no rule at all; want hidden under %v", c.prop, values) + continue + } + if hidden != c.hidden { + t.Errorf("%s under %v: hidden=%v, want %v", c.prop, values, hidden, c.hidden) + } + } +} + +func TestTimelineHideRules_AllSixRecognised(t *testing.T) { + rules, stats := extractVisibilityRulesFromJS(timelineGetProperties) + if stats.TotalHideCalls != 6 || stats.Recognized != 6 { + t.Errorf("coverage %d of %d; want 6 of 6", stats.Recognized, stats.TotalHideCalls) + } + checkTimelineCases(t, rules) +} + +// The same through the committed package (testdata/expr-checker ships Timeline +// 3.2.2), which is what `widget describe` and the .def.json generator read. +func TestTimelineHideRules_FromTheInstalledPackage(t *testing.T) { + const mpkPath = "../../testdata/expr-checker/widgets/com.mendix.widget.web.Timeline.mpk" + rules, recognized, total := ExtractWidgetVisibilityStats(mpkPath, "com.mendix.widget.web.timeline.Timeline") + if total != 6 || recognized != 6 { + t.Errorf("%d of %d editor hide-rules recognized; want 6 of 6", recognized, total) + } + checkTimelineCases(t, rules) +} + +// A switch's arms, including the shapes Timeline does not use: grouped labels, +// fall-through into the next arm, a default arm that is not last, comma and +// semicolon siblings. +func TestSwitchArms_FallThroughAndDefault(t *testing.T) { + js := `exports.getProperties=function(e,t){switch(e.k){` + + `case"a":x.hidePropertyIn(t,e,"p1");` + // falls through into b/c + `case"b":case"c":x.hidePropertyIn(t,e,"p2");break;` + + `case"d":x.hidePropertyIn(t,e,"p3"),x.hidePropertyIn(t,e,"p4");x.hidePropertyIn(t,e,"p5");break;` + + `default:x.hidePropertyIn(t,e,"p6");` + // falls through into e + `case"e":x.hidePropertyIn(t,e,"p7")` + + `}return t}` + rules, stats := extractVisibilityRulesFromJS(js) + if stats.Recognized != stats.TotalHideCalls { + t.Errorf("coverage %d of %d", stats.Recognized, stats.TotalHideCalls) + } + hiddenFor := map[string][]string{ + "p1": {"a"}, + "p2": {"a", "b", "c"}, + "p3": {"d"}, "p4": {"d"}, "p5": {"d"}, + "p6": {"z"}, // only a value no case names + "p7": {"e", "z"}, // its own case, and the default that falls into it + } + for prop, want := range hiddenFor { + for _, k := range []string{"a", "b", "c", "d", "e", "z"} { + hidden, _ := ruleHidesUnder(rules, prop, map[string]string{"k": k}) + if hidden != slices.Contains(want, k) { + t.Errorf("%s with k=%q: hidden=%v, want %v", prop, k, hidden, slices.Contains(want, k)) + } + } + } +} + +// Shapes a switch rule cannot state soundly are lifted as nothing — the safe +// direction, which is what each was before. +func TestSwitchArms_UnrepresentableShapesProduceNoRule(t *testing.T) { + cases := map[string]string{ + // the arm's hide carries its own guard: hidden on `k=="a" AND flag` + "q1": `function(e,t){switch(e.k){case"a":e.flag&&x.hidePropertyIn(t,e,"q1");break}return t}`, + // a label that is not a literal makes the default arm's complement unknown + "q2": `function(e,t){switch(e.k){case n:x.hidePropertyIn(t,e,"q0");break;default:x.hidePropertyIn(t,e,"q2")}return t}`, + // a switch under an `if` runs only when the if holds + "q3": `function(e,t){if(e.on){switch(e.k){case"a":x.hidePropertyIn(t,e,"q3")}}return t}`, + // the discriminant is not one of the widget's properties (the platform) + "q4": `function(e,t,r){switch(r){case"web":x.hidePropertyIn(t,e,"q4")}return t}`, + // an arm every value reaches is unconditional, which no condition states + "q5": `function(e,t){switch(e.k){default:x.hidePropertyIn(t,e,"q5")}return t}`, + // whether the arm before falls through cannot be read off its last + // statement — a block, a conditional break — so q6's arm is not known + "q6": `function(e,t){switch(e.k){case"a":{x.hidePropertyIn(t,e,"q0");break}case"b":x.hidePropertyIn(t,e,"q6")}return t}`, + "q7": `function(e,t){switch(e.k){case"a":if(e.f)break;x.hidePropertyIn(t,e,"q0");case"b":x.hidePropertyIn(t,e,"q7")}return t}`, + } + for prop, js := range cases { + rules, _ := extractVisibilityRulesFromJS(js) + if r := findRule(rules, prop); r != nil { + t.Errorf("%s: lifted %+v from %s; want no rule", prop, r, js) + } + } +} + +// A switch inside an object list's forEach reads the item. +func TestSwitchArms_NestedItemScope(t *testing.T) { + js := `exports.getProperties=function(e,t){e.items.forEach((function(r,n){switch(r.mode){` + + `case"a":x.hideNestedPropertiesIn(t,e,"items",n,["s1"]);break;` + + `default:x.hidePropertyIn(t,e,"items",n,"s2")}}));return t}` + rules, _ := extractVisibilityRulesFromJS(js) + s1, s2 := findRule(rules, "s1"), findRule(rules, "s2") + if s1 == nil || s1.ListPropertyKey != "items" || s1.HiddenWhen == nil || + *s1.HiddenWhen != (types.WidgetVisibilityCondition{PropertyKey: "mode", Operator: "eq", Value: "a", Scope: types.ConditionScopeItem}) { + t.Errorf("s1 = %+v; want items/s1 hidden when the item's mode is \"a\"", s1) + } + if s2 == nil || s2.ListPropertyKey != "items" || s2.HiddenWhen == nil || + *s2.HiddenWhen != (types.WidgetVisibilityCondition{PropertyKey: "mode", Operator: "ne", Value: "a", Scope: types.ConditionScopeItem}) { + t.Errorf("s2 = %+v; want items/s2 hidden when the item's mode is not \"a\"", s2) + } +} + +// terser folds statements into an `if(` head the same way it does a `switch(`; +// a CALL's paren is different, because the call can itself be conditional. +func TestStatementHeadParenIsNotAGroup(t *testing.T) { + js := `exports.getProperties=function(e,t){` + + `if(e.a?x.hidePropertyIn(t,e,"r1"):x.hidePropertyIn(t,e,"r2"),e.b)return t;` + + `e.y&&foo(e.x&&x.hidePropertyIn(t,e,"r3"));return t}` + rules, _ := extractVisibilityRulesFromJS(js) + if h, _ := ruleHidesUnder(rules, "r1", map[string]string{"a": "true"}); !h { + t.Error("r1: want hidden when a is set") + } + if h, _ := ruleHidesUnder(rules, "r1", map[string]string{"a": "false"}); h { + t.Error("r1: hidden when a is not set") + } + if h, _ := ruleHidesUnder(rules, "r2", map[string]string{"a": "false"}); !h { + t.Error("r2 (the else branch, the control): want hidden when a is not set") + } + if r := findRule(rules, "r3"); r != nil { + t.Errorf("r3: lifted %+v; a guard inside a call's arguments is one conjunct of `e.y && …`", r) + } +} + +// fileUploaderGetProperties is File Uploader 2.5.0's getProperties, verbatim from +// the marketplace package (content 235351). +const fileUploaderGetProperties = `exports.getProperties=function(e,t){return e.readOnlyMode?A.hidePropertiesIn(t,e,["files"===e.uploadMode?"associatedImages":"associatedFiles","createImageAction","createFileAction","allowedFileFormats","maxFilesPerUpload","maxFilesPerBatch","maxFileSize","objectCreationTimeout"]):"files"===e.uploadMode?(A.hidePropertiesIn(t,e,["associatedImages","createImageAction"]),function(e,t){var r;e.allowedFileFormats.forEach(function(r,n){"simple"===r.configMode?A.hideNestedPropertiesIn(t,e,"allowedFileFormats",n,["mimeType","extensions","typeFormatDescription"]):A.hideNestedPropertiesIn(t,e,"allowedFileFormats",n,["predefinedType"])});var n=null===(r=t[0].properties)||void 0===r?void 0:r.find(function(e){return"allowedFileFormats"===e.key});if(n){var o;n.objectHeaders=["File type",""],null===(o=n.objects)||void 0===o||o.forEach(function(t,r){var n=e.allowedFileFormats[r];"simple"===n.configMode?t.captions=[E[n.predefinedType].description," "]:t.captions=[n.typeFormatDescription,"".concat(z(n.mimeType,n.extensions))]})}}(e,t)):A.hidePropertiesIn(t,e,["associatedFiles","createFileAction","allowedFileFormats"]),"files"===e.uploadMode?A.hidePropertiesIn(t,e,["onUploadSuccessImage","onUploadFailureImage"]):A.hidePropertiesIn(t,e,["onUploadSuccessFile","onUploadFailureFile"]),e.enableCustomButtons?e.customButtons.forEach(function(r,n){A.hideNestedPropertiesIn(t,e,"customButtons",n,"files"===e.uploadMode?["buttonActionImage"]:["buttonActionFile"])}):A.hidePropertiesIn(t,e,["customButtons"]),t};` + +// ako/mxcli#258 item 1: the ternary-wrapped `hidePropertiesIn` that gates the +// create* actions on uploadMode. It was unread when filed ("6 of 9"); the +// else-if chain reading added for mendixlabs/mxcli#1583 lifts it, so this pins +// that, and that the switch/statement-head changes for #1762 leave it alone. +func TestFileUploaderUploadModeGate_258(t *testing.T) { + rules, _ := extractVisibilityRulesFromJS(fileUploaderGetProperties) + cases := []struct { + prop, readOnly, mode string + hidden bool + }{ + {"createImageAction", "false", "files", true}, + {"createImageAction", "false", "images", false}, + {"createImageAction", "true", "images", true}, + {"createFileAction", "false", "images", true}, + {"createFileAction", "false", "files", false}, + {"createFileAction", "true", "files", true}, + {"onUploadSuccessImage", "false", "files", true}, + {"onUploadSuccessImage", "false", "images", false}, + {"onUploadSuccessFile", "false", "images", true}, + {"onUploadSuccessFile", "false", "files", false}, + } + for _, c := range cases { + values := map[string]string{"readOnlyMode": c.readOnly, "uploadMode": c.mode} + if hidden, _ := ruleHidesUnder(rules, c.prop, values); hidden != c.hidden { + t.Errorf("%s under %v: hidden=%v, want %v", c.prop, values, hidden, c.hidden) + } + } +} diff --git a/mdl/executor/validate_timeline_hidden_1762_test.go b/mdl/executor/validate_timeline_hidden_1762_test.go new file mode 100644 index 000000000..b5e4302ee --- /dev/null +++ b/mdl/executor/validate_timeline_hidden_1762_test.go @@ -0,0 +1,136 @@ +// SPDX-License-Identifier: Apache-2.0 + +// A Timeline enumeration its editorConfig hides, set to a non-default value +// (ako/mxcli#1762, mendixlabs/mxcli#1668). Reported: `mxcli check` and `exec` +// both pass, and mx check reports "[CE0463] The definition of this widget has +// changed" at Timeline 'tlD' and 'tlG'. +// +// The check already refused this shape for the one rule it recognised — +// `groupEvents: false` plus a non-default hidden value is an MDL-WIDGET10 +// error, "a non-default value there fails the build with CE0463" — so the +// remaining rules get the same verdict rather than a second policy: refused +// when the script names the value, written at the default when it does not. +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// timelineVariants mirrors the measurement in editorconfig_switch_1762_test.go: +// one widget per configuration, with what mx check said about it on 11.13.0. +var timelineVariants = []struct { + name, props string + hiddenProp string // "" for a control mx check passed +}{ + {"tlB", `customVisualization: true`, ""}, + {"tlD", `customVisualization: true, groupByKey: 'day', groupByDayOptions: 'dayMonth'`, "groupByDayOptions"}, + {"tlDm", `customVisualization: true, groupByKey: 'month', groupByMonthOptions: 'monthYear'`, "groupByMonthOptions"}, + {"tlC", `groupByKey: 'day', groupByDayOptions: 'dayMonth'`, ""}, + {"tlM", `groupByKey: 'month', groupByMonthOptions: 'monthYear'`, ""}, + {"tlG", `groupByKey: 'day', groupByMonthOptions: 'monthYear'`, "groupByMonthOptions"}, + {"tlF", `groupByKey: 'month', groupByDayOptions: 'fullDate'`, "groupByDayOptions"}, + {"tlY", `groupByKey: 'year', groupByDayOptions: 'dayMonth'`, "groupByDayOptions"}, + {"tlYm", `groupByKey: 'year', groupByMonthOptions: 'monthYear'`, "groupByMonthOptions"}, +} + +func timelinePage(name, props string) string { + return "create page ModTL.P_" + name + " (Title: 'T', Layout: Atlas_Core.Atlas_Default) {\n" + + " timeline " + name + " (data: MICROFLOW ModTL.DS_Events, groupAttribute: At, " + props + ")\n};\n" +} + +func widget10For(vs []linter.Violation, widget string) []linter.Violation { + var out []linter.Violation + for _, v := range vs { + if v.RuleID == "MDL-WIDGET10" && strings.Contains(v.Message, "widget `"+widget+"`") { + out = append(out, v) + } + } + return out +} + +// check -p against the installed Timeline 3.2.2 agrees with mxbuild on every +// measured variant. +func TestTimelineHiddenEnumeration_CheckAgreesWithMxbuild(t *testing.T) { + project := projectFixture(t) + var src strings.Builder + for _, v := range timelineVariants { + src.WriteString(timelinePage(v.name, v.props)) + } + prog, errs := visitor.Build(src.String()) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + all := ValidateWidgetProperties(prog, project) + + for _, v := range timelineVariants { + got := widget10For(all, v.name) + if v.hiddenProp == "" { + for _, g := range got { + if g.Severity == linter.SeverityError { + t.Errorf("%s (mx check: 0 errors): refused with %s", v.name, g.Message) + } + } + continue + } + refused := false + for _, g := range got { + if g.Severity == linter.SeverityError && strings.Contains(g.Message, "`"+v.hiddenProp+"`") && + strings.Contains(g.Message, "CE0463") { + refused = true + } + } + if !refused { + t.Errorf("%s (mx check: CE0463): want an MDL-WIDGET10 error naming `%s`, got %+v", v.name, v.hiddenProp, got) + } + } + // The default arm's rule says which values hide it, not "matches". + if got := widget10For(all, "tlY"); len(got) == 0 || + !strings.Contains(got[0].Message, "`groupByKey` is none of \"day\", \"month\"") { + t.Errorf("tlY: message %+v; want it to name the values the default arm covers", got) + } +} + +// The value split the check already makes: a hidden enumeration set to its +// default is redundant, not a build failure, so it stays a warning. +func TestTimelineHiddenEnumeration_DefaultIsOnlyAWarning(t *testing.T) { + project := projectFixture(t) + prog, errs := visitor.Build(timelinePage("tlDef", `customVisualization: true, groupByDayOptions: 'dayName'`)) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + got := widget10For(ValidateWidgetProperties(prog, project), "tlDef") + if len(got) != 1 || got[0].Severity != linter.SeverityWarning { + t.Fatalf("got %+v; want one MDL-WIDGET10 warning (the value will be ignored)", got) + } +} + +const timelineSetup = `create module ModTL; +create non-persistent entity ModTL.Ev ( Line: String(200), At: DateTime ); +create microflow ModTL.DS_Events () returns List of ModTL.Ev as $Events +begin + $Events = create list of ModTL.Ev; + return $Events; +end; +` + +// The other half of the policy: a hidden enumeration the script does NOT name is +// written at its default (hiddenUnnamedProperties), now that the rules hiding +// it are known. exec's refusal of a named one is its pre-flight running this +// same check — see cmd/mxcli/exec_preflight_timeline_1762_test.go. +func TestTimelineHiddenEnumeration_UnnamedOnesAreWrittenAtTheirDefault(t *testing.T) { + exec, _ := pageFixtureExecutor(t) + run(t, exec, timelineSetup) + if err := runProgErr(t, exec, timelinePage("tlB", `customVisualization: true`)); err != nil { + t.Fatalf("exec of the variant mx check passed: %v", err) + } + id := unitIDByName(t, exec, "page", "P_tlB") + for key, want := range map[string]string{"groupByDayOptions": "dayName", "groupByMonthOptions": "month"} { + if got := storedPrimitive854(t, exec, id, "tlB", key); got != want { + t.Errorf("%s stored %q; a hidden enumeration must be stored at its default %q", key, got, want) + } + } +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index 8b82c3878..7f3d8737c 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -939,11 +939,30 @@ func visibilityCondWord(c *types.WidgetVisibilityCondition) string { return "is enabled" case "falsy": return "is disabled" + case "empty": + return "is empty" + case "notempty": + return "is not empty" + case "in": + return "is one of " + quotedCSV(c.Value) + case "notin": + // A switch's default arm (Timeline's `groupByKey` other than day/month). + return "is none of " + quotedCSV(c.Value) default: return "matches" } } +// quotedCSV renders an `in`/`notin` condition's comma-separated values as a +// quoted, comma-separated list. +func quotedCSV(csv string) string { + parts := strings.Split(csv, ",") + for i, p := range parts { + parts[i] = fmt.Sprintf("%q", p) + } + return strings.Join(parts, ", ") +} + // validateObjectListItemEnums flags an enumeration sub-property of an object-list // item (e.g. a Maps marker `LocationType`, a chart series `Interpolation`) whose // value isn't one of the widget's declared member keys. Studio Pro silently diff --git a/mdl/executor/widget_defs.go b/mdl/executor/widget_defs.go index 3aca88263..f2841c8fa 100644 --- a/mdl/executor/widget_defs.go +++ b/mdl/executor/widget_defs.go @@ -95,9 +95,8 @@ func RefreshWidgetDefinitions(projectPath string, force bool, output io.Writer) // Lift property-visibility rules from the widget's editorConfig.js // (#574 Phase 2) so the generated .def.json carries the version- // specific applicability logic. Merge with the hand-transcribed table - // rather than replace it: the static extractor skips compound/ternary - // guards (e.g. Timeline's `customVisualization ? hide([title,...])`), - // so the hand-authored fallback fills the keys the extractor misses. + // rather than replace it: the static extractor skips some guard + // shapes, so the hand-authored fallback fills the keys it misses. // Extracted rules win on conflict (they're version-specific). if rules := extractVisibilityRulesFromMPK(mpkPath, mpkDef.ID); len(rules) > 0 { defJSON.PropertyVisibility = mergeVisibilityRules(rules, widgetVisibilityRules[mpkDef.ID]) @@ -442,20 +441,26 @@ func GenerateDefJSON(mpkDef *mpk.WidgetDefinition, mdlName string) *WidgetDefini // configurations. Until the JS extractor lands (#574 Phase 2), these are // transcribed by hand from each widget's compiled editorConfig.js. // -// Only TextTemplate-typed hidden properties need entries: the populated-vs-null -// ClientTemplate choice is what triggers CE0463. Properties hidden as -// Expression/enum/Widgets slots don't carry a ClientTemplate and are omitted. +// The entries cover TextTemplate properties only, and that is NOT a statement +// that only TextTemplates matter. Any hidden property must hold its default: an +// enumeration hidden at a non-default value is CE0463 just the same (Timeline's +// groupByDayOptions in custom mode, ako/mxcli#1762). This table once made that +// rule look recognised — `check` warned about the Timeline's hidden templates, +// from here, while the enumerations in the very same hide went unchecked. // // VideoPlayer (editorConfig.js): // "expression"===e.type && hidePropertiesIn(["videoUrl","posterUrl"]) // Timeline (editorConfig.js): // e.customVisualization ? hidePropertiesIn(["title","description","icon","timeIndication",...]) : ... // +// The extractor now lifts the whole Timeline hide (3.1.1, 3.2.2 and 3.2.3 all +// read 6 of 6), so its entry only matters for a package that ships no +// editorConfig. +// // mergeVisibilityRules returns the extracted rules plus any hand-authored fallback // rules whose PropertyKey the extractor did not cover. Extracted rules are // version-specific (lifted from the installed .mpk's editorConfig.js) and win on -// conflict; the fallback fills the compound/ternary guards the static extractor -// skips (e.g. Timeline `title`/`description` hidden when `customVisualization`). +// conflict; the fallback fills guards the static extractor skips. func mergeVisibilityRules(extracted, fallback []types.WidgetVisibilityRule) []types.WidgetVisibilityRule { if len(fallback) == 0 { return extracted diff --git a/mdl/executor/widget_engine.go b/mdl/executor/widget_engine.go index d008a6bcd..ae79fc557 100644 --- a/mdl/executor/widget_engine.go +++ b/mdl/executor/widget_engine.go @@ -100,7 +100,12 @@ const defaultSlotContainer = "template" // `editable:` when the package declares no Editability); before, it was // known-but-unmapped and the Calendar's `editable` could not be written // (mendixlabs/mxcli#1578). -const WidgetDefGeneratorVersion = 19 +// 20 — lift editorConfig hides in a `switch` arm, and a guard bounded by the +// head of a `switch(`/`if(` statement (terser folds earlier statements +// into it). Timeline went from 2 to 6 of 6 rules, so its hidden +// enumerations are refused with MDL-WIDGET10 rather than reaching mx check +// as CE0463 (ako/mxcli#1762). Bump forces existing projects to regenerate. +const WidgetDefGeneratorVersion = 20 // WidgetDefinition describes how to construct a pluggable widget from MDL syntax. // Loaded from embedded JSON definition files (*.def.json). From afe48e67c72c17c8e8e281d73ac844cde5b2e861 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 23:02:50 +0000 Subject: [PATCH 20/44] fix(describe): leave out a pluggable widget property it hides at its default Once the Timeline's hide rules are read (ako/mxcli#1762), one of its two format enumerations is always hidden: by customVisualization, or by the groupByKey switch. The generic pluggable-widget describe printed every stored primitive, so each Timeline came back with a hidden property at its default and drew an MDL-WIDGET10 warning on mxcli's own output (11 warnings on six measured pages). The same happened before for any widget hiding an enumeration by a rule the extractor already read, such as a Timeline with `groupEvents: false`. Such a property carries nothing: exec writes a hidden property the script leaves out at that same default. extractExplicitProperties now drops a top-level property that the widget's own rules hide in the stored configuration while it equals the stored schema's default. The rules are the registry's, the ones `check` evaluates, so the two agree about what is hidden. A hidden property holding a non-default value is still printed, because that widget fails mx check with CE0463, and printing it lets `check` say so. describe -> check -> exec on three Timeline configurations: no MDL-WIDGET10, and the stored page is byte-identical afterwards. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...meline-hidden-enumeration-switch-arms.json | 2 +- CHANGELOG.md | 2 +- docs-site/src/guides/pluggable-widgets.md | 2 +- mdl/executor/cmd_pages_describe_pluggable.go | 56 ++++++++++++++- ...ages_describe_timeline_hidden_1762_test.go | 70 +++++++++++++++++++ 5 files changed, 128 insertions(+), 4 deletions(-) create mode 100644 mdl/executor/cmd_pages_describe_timeline_hidden_1762_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json index e8d3f6949..ee7743e72 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-28-timeline-hidden-enumeration-switch-arms.json @@ -1 +1 @@ -{"area":"mdl/executor","date":"2026-10-28","symptom":"A Timeline enumeration the widget hides, set to a non-default value — `customVisualization: true, groupByDayOptions: 'dayMonth'`, or `groupByKey: 'day', groupByMonthOptions: 'monthYear'` — passes `mxcli check` and `exec`, and mx check reports \"[CE0463] The definition of this widget has changed\" at Timeline 'tlD' / 'tlG'. `mxcli widget describe timeline` says \"2 of 6 editor hide-rules recognized\"; `groupEvents: false` plus a hidden value was already an MDL-WIDGET10 error (ako/mxcli#1762, mendixlabs/mxcli#1668)","cause":"Two editorConfig shapes went unread. terser folds the statements before a `switch` into its head — `switch(e.customVisualization ? hide([…]) : hide([…]), e.groupEvents || hide([…]), e.groupByKey)` — so the THEN branch's guard is bounded by the `switch(` paren, which parseGuard rejected as a conditional group (the else branch was lifted, through trailingExpr). And a hide in a `case`/`default` arm had no reading at all: its `case \"day\":` colon reached the ternary-else path and found no `?`. The template warnings the reporter saw in custom mode came from the hand-written fallback table in widget_defs.go, which listed only the TextTemplate keys of that same hide, on the stated premise that enumerations cannot cause CE0463","file":"mdl/executor/editorconfig_switch.go (switchArmCondition, switchArms, statementHeadParen, atFunctionStatementLevel), editorconfig_extract.go (the switch-arm branch before parseGuard; parseGuard's `(` boundary), validate_widgets.go (visibilityCondWord in/notin), widget_engine.go (WidgetDefGeneratorVersion 20)","insight":"**A known rule elsewhere is not the extractor knowing it.** The custom-mode warning made rule (1) look recognised, and it was the reason to ask why the enumerations in the same list were missed: `widget describe` (extractor only) showed no `title` rule, while the generated .def.json had one, merged from the hand table. Compare the two sources first when a rule half-works. **The reporter's framing (`the true branch of a ternary`) was not the cause**: the existing TimelinePattern test already lifted a statement-level THEN branch, and only the minified file showed that terser had moved it into the `switch(` head. Read the shipped JS, not the de-minified source. **Measure the matrix, then make it the oracle**: one widget per variant through `mx check --no-update-widgets` on 11.13.0 gave 6 CE0463 and 3 clean, and the extractor test, the check test and the bug-test all encode those nine. **Policy follows the existing split, not a second one**: hidden + non-default is refused (MDL-WIDGET10 error, exec preflight) and hidden + left out is written at the default, as rule (3) and the column-SET reset (mendixlabs/mxcli#1519) already do. **Bound the blast radius by dumping lifted rules for every installed .mpk before and after**: only Timeline and HeatMap changed. HeatMap folds into an `if(` head the same way, and its new rule is correct. **The scratchpad can be shared between agents**: work in a private subdirectory","refs":["ako/mxcli#1762","mendixlabs/mxcli#1668","ako/mxcli#258","mendixlabs/mxcli#1583"],"ce":["CE0463"],"rules":["MDL-WIDGET10"],"test":"mdl/executor/editorconfig_switch_1762_test.go; validate_timeline_hidden_1762_test.go; cmd/mxcli/exec_preflight_timeline_1762_test.go; mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl"} +{"area":"mdl/executor","date":"2026-10-28","symptom":"A Timeline enumeration the widget hides, set to a non-default value — `customVisualization: true, groupByDayOptions: 'dayMonth'`, or `groupByKey: 'day', groupByMonthOptions: 'monthYear'` — passes `mxcli check` and `exec`, and mx check reports \"[CE0463] The definition of this widget has changed\" at Timeline 'tlD' / 'tlG'. `mxcli widget describe timeline` says \"2 of 6 editor hide-rules recognized\"; `groupEvents: false` plus a hidden value was already an MDL-WIDGET10 error (ako/mxcli#1762, mendixlabs/mxcli#1668)","cause":"Two editorConfig shapes went unread. terser folds the statements before a `switch` into its head — `switch(e.customVisualization ? hide([…]) : hide([…]), e.groupEvents || hide([…]), e.groupByKey)` — so the THEN branch's guard is bounded by the `switch(` paren, which parseGuard rejected as a conditional group (the else branch was lifted, through trailingExpr). And a hide in a `case`/`default` arm had no reading at all: its `case \"day\":` colon reached the ternary-else path and found no `?`. The template warnings the reporter saw in custom mode came from the hand-written fallback table in widget_defs.go, which listed only the TextTemplate keys of that same hide, on the stated premise that enumerations cannot cause CE0463","file":"mdl/executor/editorconfig_switch.go (switchArmCondition, switchArms, statementHeadParen, atFunctionStatementLevel), editorconfig_extract.go (the switch-arm branch before parseGuard; parseGuard's `(` boundary), validate_widgets.go (visibilityCondWord in/notin), cmd_pages_describe_pluggable.go (withoutHiddenDefaults), widget_engine.go (WidgetDefGeneratorVersion 20)","insight":"**A known rule elsewhere is not the extractor knowing it.** The custom-mode warning made rule (1) look recognised, and it was the reason to ask why the enumerations in the same list were missed: `widget describe` (extractor only) showed no `title` rule, while the generated .def.json had one, merged from the hand table. Compare the two sources first when a rule half-works. **The reporter's framing (`the true branch of a ternary`) was not the cause**: the existing TimelinePattern test already lifted a statement-level THEN branch, and only the minified file showed that terser had moved it into the `switch(` head. Read the shipped JS, not the de-minified source. **Measure the matrix, then make it the oracle**: one widget per variant through `mx check --no-update-widgets` on 11.13.0 gave 6 CE0463 and 3 clean, and the extractor test, the check test and the bug-test all encode those nine. **Policy follows the existing split, not a second one**: hidden + non-default is refused (MDL-WIDGET10 error, exec preflight) and hidden + left out is written at the default, as rule (3) and the column-SET reset (mendixlabs/mxcli#1519) already do. **Knowing more rules made describe noisy**: every Timeline hides one of its two formats, and the generic describe printed every stored primitive, so its own output drew MDL-WIDGET10 warnings. describe now drops a hidden property at its default, using the same registry rules the check reads. **Bound the blast radius by dumping lifted rules for every installed .mpk before and after**: only Timeline and HeatMap changed. HeatMap folds into an `if(` head the same way, and its new rule is correct. **The scratchpad can be shared between agents**: work in a private subdirectory","refs":["ako/mxcli#1762","mendixlabs/mxcli#1668","ako/mxcli#258","mendixlabs/mxcli#1583"],"ce":["CE0463"],"rules":["MDL-WIDGET10"],"test":"mdl/executor/editorconfig_switch_1762_test.go; validate_timeline_hidden_1762_test.go; cmd_pages_describe_timeline_hidden_1762_test.go; cmd/mxcli/exec_preflight_timeline_1762_test.go; mdl-examples/bug-tests/1762-timeline-hidden-enumeration.mdl"} diff --git a/CHANGELOG.md b/CHANGELOG.md index c33507cc8..b7a9cfb96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed -- **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. +- **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/docs-site/src/guides/pluggable-widgets.md b/docs-site/src/guides/pluggable-widgets.md index 6ef086397..6d0812a4d 100644 --- a/docs-site/src/guides/pluggable-widgets.md +++ b/docs-site/src/guides/pluggable-widgets.md @@ -55,7 +55,7 @@ version you actually have at the moment you create the widget. So a hidden property you leave out is written at its default; one you set to its default draws an MDL-WIDGET10 warning (the value is ignored); and one you set to anything else is an MDL-WIDGET10 **error**, which `exec` refuses before writing. - For example, a + `describe` leaves out a hidden property that holds its default. For example, a Timeline grouped by `'day'` hides `groupByMonthOptions`, and one with `customVisualization: true` hides both format enumerations. diff --git a/mdl/executor/cmd_pages_describe_pluggable.go b/mdl/executor/cmd_pages_describe_pluggable.go index 91eed9d63..7c1579073 100644 --- a/mdl/executor/cmd_pages_describe_pluggable.go +++ b/mdl/executor/cmd_pages_describe_pluggable.go @@ -6,6 +6,8 @@ import ( "context" "strings" "sync" + + "github.com/mendixlabs/mxcli/mdl/types" ) // buildPropertyTypeKeyMap builds a map from PropertyType $ID to PropertyKey for a CustomWidget. @@ -1320,6 +1322,8 @@ func extractExplicitProperties(ctx *ExecContext, w map[string]any, enclosingEnti } var result []rawExplicitProp + stored := map[string]string{} // primitive values, for the visibility rules + defaults := map[string]string{} // the stored schema's DefaultValue per key props := getBsonArrayElements(obj["Properties"]) for _, prop := range props { propMap, ok := prop.(map[string]any) @@ -1335,6 +1339,10 @@ func extractExplicitProperties(ctx *ExecContext, w map[string]any, enclosingEnti if !ok { continue } + if pv := extractString(value["PrimitiveValue"]); pv != "" { + stored[propKey] = pv + } + defaults[propKey] = schemas[typePointerID].Default // Check for AttributeRef (attribute binding) if attrRef, ok := value["AttributeRef"].(map[string]any); ok && attrRef != nil { @@ -1401,7 +1409,53 @@ func extractExplicitProperties(ctx *ExecContext, w map[string]any, enclosingEnti }) } } - return result + return withoutHiddenDefaults(ctx, extractCustomWidgetID(ctx, w), result, stored, defaults) +} + +// withoutHiddenDefaults drops the described properties the widget's own +// editorConfig hides in its stored configuration while they hold their +// default. Such a property says nothing — exec writes a hidden property the +// script leaves out at that same default (hiddenUnnamedProperties) — and +// printing it drew an MDL-WIDGET10 warning on describe's own output: every +// Timeline hides one of its two format enumerations (ako/mxcli#1762). +// +// The rules are the ones `check` evaluates (the project's widget registry), so +// the two cannot disagree about what is hidden. A hidden property holding a +// NON-default value is kept: that widget fails mx check with CE0463, and +// printing the value is what lets `check` say so instead of describe hiding it. +func withoutHiddenDefaults(ctx *ExecContext, widgetID string, props []rawExplicitProp, stored, defaults map[string]string) []rawExplicitProp { + reg := ctx.GetWidgetRegistry() + if reg == nil || widgetID == "" || len(props) == 0 { + return props + } + def, ok := reg.GetByWidgetID(widgetID) + if !ok { + return props + } + hidden := map[string]bool{} + for _, rule := range visibilityRulesFor(def, reg) { + if rule.Nested() || rule.HiddenWhen == nil { + continue + } + fires, determinable := rule.Fires(func(c types.WidgetVisibilityCondition) (string, bool) { + v, ok := stored[c.PropertyKey] + return v, ok + }) + if determinable && fires { + hidden[rule.PropertyKey] = true + } + } + if len(hidden) == 0 { + return props + } + out := props[:0:0] + for _, p := range props { + if hidden[p.Key] && !p.IsRef && len(p.Params) == 0 && defaults[p.Key] != "" && p.Value == defaults[p.Key] { + continue + } + out = append(out, p) + } + return out } // extractImageProperties extracts properties from a pluggable Image CustomWidget. diff --git a/mdl/executor/cmd_pages_describe_timeline_hidden_1762_test.go b/mdl/executor/cmd_pages_describe_timeline_hidden_1762_test.go new file mode 100644 index 000000000..a46215405 --- /dev/null +++ b/mdl/executor/cmd_pages_describe_timeline_hidden_1762_test.go @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" +) + +// describe → check → exec on a Timeline, once the hide rules are known +// (ako/mxcli#1762). One of the two format enumerations is ALWAYS hidden — by +// customVisualization, or by the groupByKey switch — and a generic pluggable +// widget was described with every stored primitive, so each Timeline came back +// with a hidden property at its default and drew an MDL-WIDGET10 warning on +// mxcli's own output. Such a property carries nothing: exec writes a hidden +// property the script does not name at that same default. +func TestTimelineDescribe_HiddenDefaultsAreNotPrinted(t *testing.T) { + exec, out := pageFixtureExecutor(t) + run(t, exec, timelineSetup) + variants := []struct{ name, props, mustPrint, mustNotPrint string }{ + {"tlB", `customVisualization: true`, "customVisualization", "groupByDayOptions"}, + {"tlS", `groupByKey: 'year'`, "groupByKey: 'year'", "groupByMonthOptions"}, + {"tlM", `groupByKey: 'month', groupByMonthOptions: 'monthYear'`, "groupByMonthOptions: 'monthYear'", "groupByDayOptions"}, + } + for _, v := range variants { + run(t, exec, timelinePage(v.name, v.props)) + } + for _, v := range variants { + id := unitIDByName(t, exec, "page", "P_"+v.name) + before := storedBytes(t, exec, id) + + out.Reset() + run(t, exec, "describe page ModTL.P_"+v.name) + described := out.String() + if !strings.Contains(described, v.mustPrint) { + t.Errorf("%s: describe lost %q:\n%s", v.name, v.mustPrint, described) + } + if strings.Contains(described, v.mustNotPrint) { + t.Errorf("%s: describe printed the hidden %s:\n%s", v.name, v.mustNotPrint, described) + } + + prog, errs := visitor.Build(described) + if len(errs) > 0 { + t.Fatalf("%s: describe output does not parse: %v\n%s", v.name, errs, described) + } + if got := widget10For(ValidateWidgetProperties(prog, exec.backend.Path()), v.name); len(got) > 0 { + t.Errorf("%s: check of describe's own output reports %+v", v.name, got) + } + + if err := runErr(t, exec, described); err != nil { + t.Fatalf("%s: exec of the description: %v", v.name, err) + } + if !bytes.Equal(before, storedBytes(t, exec, id)) { + t.Errorf("%s: describe → exec changed the stored page", v.name) + } + } +} + +func storedBytes(t *testing.T, exec *Executor, unitID string) []byte { + t.Helper() + raw, err := exec.backend.GetRawUnitBytes(model.ID(unitID)) + if err != nil { + t.Fatal(err) + } + return append([]byte(nil), raw...) +} From 75ee5480565abcf16f8e2f075c4ffe6f835ae9d4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 23:16:27 +0000 Subject: [PATCH 21/44] fix(microflows): edit an activity replaced by one of the same kind in place Re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new $IDs, while a decision whose condition changed the same way kept its own, and exec reported both as `spliced: 1 replaced`. Studio Pro compares and merges by $ID, so it showed the edit as one activity removed and another added (ako/mxcli#1761, mendixlabs/mxcli#1667). A splice Replace by one activity of the same activity and action $Type now edits the stored activity: the replacement is written into the stored slot, at the stored size, and its elements are paired structurally with the stored ones (same $Type at the same key; list entries by the member, parameter or language they name; unnamed entries by position only for equal $Type sequences). The stored $IDs go on at Bytes, rewriting every reference in the same pass, so the executor can still name the fragment's activity until then. This applies at top level, in a loop body and to an activity with a custom error handler, for create or modify and for alter ... replace. A run of adjacent statements each changed in its own place, pairwise of one plain kind, is diffed one-for-one instead of as a replace-and-drop of the whole run, so each is edited in place. The report tells the two outcomes apart: `edited in place` for a node that kept its $ID (an activity of the same kind, a decision's condition, a return's value, a node's notes) and `replaced with new element IDs` for one that did not. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...e-kind-activity-replace-mints-new-ids.json | 1 + CHANGELOG.md | 1 + ...create-or-modify-edit-inside-loop-body.mdl | 5 +- .../1489-error-handler-activity-splices.mdl | 4 +- ...notes-on-decision-and-end-event-splice.mdl | 2 +- .../1610-decision-caption-change-splices.mdl | 2 +- ...1761-same-kind-activity-edit-keeps-ids.mdl | 64 +++ ...e-856-list-operation-where-bare-member.mdl | 4 +- mdl/backend/mfmutator/splice.go | 54 ++- mdl/backend/mfmutator/splice_edit.go | 332 +++++++++++++ .../mfmutator/splice_edit_in_place_test.go | 441 ++++++++++++++++++ mdl/backend/mfmutator/splice_handler_test.go | 92 ++-- mdl/backend/microflow_mutation.go | 10 +- mdl/executor/cmd_alter_flow.go | 24 +- .../cmd_flow_loop_body_edit_pedapp_test.go | 18 +- mdl/executor/cmd_flow_modify.go | 59 ++- .../flow_splice_decision_1694_pedapp_test.go | 12 +- ...w_splice_edit_in_place_1761_pedapp_test.go | 364 +++++++++++++++ .../flow_splice_error_handler_pedapp_test.go | 42 +- ...ow_splice_list_op_where_856_pedapp_test.go | 17 +- .../flow_splice_loop_body_1693_pedapp_test.go | 2 +- .../flow_commit_events_rerun_test.go | 4 +- mdl/roundtrip/flow_idempotent_shapes_test.go | 4 +- mdl/roundtrip/flow_modify_end_test.go | 4 +- mdl/roundtrip/flow_modify_notes_test.go | 4 +- mdl/roundtrip/flow_modify_test.go | 12 +- mdl/roundtrip/flow_respelling_rerun_test.go | 16 +- mdl/roundtrip/flow_splice_guard_test.go | 6 +- .../flow_splice_handler_return_test.go | 2 +- .../flow_splice_member_qualified_test.go | 4 +- mdl/roundtrip/flow_splice_parity_test.go | 4 +- mdl/roundtrip/rerun_idempotent_test.go | 2 +- 32 files changed, 1476 insertions(+), 136 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-backend/2026-10-28-same-kind-activity-replace-mints-new-ids.json create mode 100644 mdl-examples/bug-tests/1761-same-kind-activity-edit-keeps-ids.mdl create mode 100644 mdl/backend/mfmutator/splice_edit.go create mode 100644 mdl/backend/mfmutator/splice_edit_in_place_test.go create mode 100644 mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend/2026-10-28-same-kind-activity-replace-mints-new-ids.json b/.claude/skills/fix-issue/findings/mdl-backend/2026-10-28-same-kind-activity-replace-mints-new-ids.json new file mode 100644 index 000000000..576a61b5a --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-backend/2026-10-28-same-kind-activity-replace-mints-new-ids.json @@ -0,0 +1 @@ +{"area":"mdl/backend","date":"2026-10-28","symptom":"Re-running `create or modify microflow|nanoflow` with only a filter's expression changed gave the activity, its ListOperationsAction and its FilterByExpression new $IDs, while a decision whose condition changed the same way kept its own; exec printed `spliced: 1 replaced` for both, and check, mx check, diff and describe were clean, so only a BSON dump or Studio Pro's version control (one activity removed, another added) showed it","cause":"The statement diff turns a changed activity into a Replace, and Mutator.Replace at top level appended the fragment's freshly built activity and removed the stored one. The in-place swaps that existed (swapInPlace: loop bodies #1106, handled activities #1489) kept the stored slot and geometry but still wrote the fragment's $IDs. The patch write (UpdateRawUnitPatch) deliberately skips canon.TransplantIDs, so nothing downstream re-paired them. Separately, two adjacent changed activities form one run, which became 'replace the first by both, drop the second' and could never be an edit","file":"mdl/backend/mfmutator/splice_edit.go (sameKind, swapKeeping, editInPlace, carryPairs, pairEntries, applyCarried), mdl/backend/mfmutator/splice.go (Replace, replaceInLoop, replaceKeepingHandler, Bytes), mdl/executor/cmd_alter_flow.go (noteEdited), mdl/executor/cmd_flow_modify.go (patchSummary, pairwiseSameKind)","fix":"A replace by one activity of the same activity and action $Type is written into the stored slot and paired structurally with the stored activity: same $Type at the same key, list entries by what they name (Attribute/Association plus Type, Parameter, LanguageCode, Name, Key), unnamed entries by position only for equal $Type sequences. The pairs are applied at Bytes over every binary in the unit, so the executor can keep naming the fragment's $ID (LinkNote of a stored note) until then. MicroflowMutator.EditedInPlace feeds the report: `edited in place` vs `replaced with new element IDs`. A run of N changed statements pairwise of one plain kind is split into N one-for-one gaps","test":"mdl/backend/mfmutator/splice_edit_in_place_test.go (six kinds byte-exact, members by name, another kind/operation is new, notes, loop body, error handler); mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go (six kinds x microflow/nanoflow on the real codec, a control that a log for a filter is new, and PedApp's Studio Pro-authored PopulateUserAttributes edited in one run). With the carry stubbed out the engine and executor tests fail with new $IDs; measured on 10.24.24: unfixed 1ccaa1aa->1c24b792 for the activity, fixed all kept, mx check 0 errors","insight":"The identity carry has to happen where the pairing is known, and that is the splice, not canon: the patch writer is told the splice owns its $IDs (ContentsOwnElementIDs), so a fix that hopes Reconcile will re-pair finds it switched off on purpose. Pair by name before position — a member list paired by index hands a removed member's $ID to its neighbour, which is the GUID-class mistake one level down. And apply the carry late: the executor still names the fragment's $IDs after Replace returns (linkStoredNotes draws a stored note's line to the fragment's activity), so the stored $IDs go on at Bytes, where one pass rewrites every reference too. The report half matters as much: a count that prints 'replaced' for both outcomes is why the bug read as harmless; the test that a decision 'kept its IDs' and a filter did not was only visible in a BSON dump","refs":["ako/mxcli#1761","mendixlabs/mxcli#1667","ako/mxcli#1106","mendixlabs/mxcli#1489","ako/mxcli#888"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index b7a9cfb96..9734de71e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. +- **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/mdl-examples/bug-tests/1106-create-or-modify-edit-inside-loop-body.mdl b/mdl-examples/bug-tests/1106-create-or-modify-edit-inside-loop-body.mdl index c3df1a7a6..4fb535f2b 100644 --- a/mdl-examples/bug-tests/1106-create-or-modify-edit-inside-loop-body.mdl +++ b/mdl-examples/bug-tests/1106-create-or-modify-edit-inside-loop-body.mdl @@ -14,9 +14,10 @@ -- To reproduce: exec this file against a project, then change -- `formatDecimal($It/Amount, '0.#')` to `toString(round($It/Amount, 1))` and -- exec it again. The second run now reports --- Modified microflow: BugTest1106.BuildJson (spliced: 1 replaced) +-- Modified microflow: BugTest1106.BuildJson (spliced: 1 edited in place) -- and a third run writes nothing. The loop and the other activities keep their --- $IDs and positions; only the edited `set` is replaced, in place. +-- $IDs and positions; the edited `set` is the stored activity with its new +-- expression, and keeps its $ID too (ako/mxcli#1761). -- -- Still refused (needs the body laid out again): a statement added, removed, -- reordered or moved inside the body, or one kind of statement swapped for diff --git a/mdl-examples/bug-tests/1489-error-handler-activity-splices.mdl b/mdl-examples/bug-tests/1489-error-handler-activity-splices.mdl index 26da19f0c..712d8f341 100644 --- a/mdl-examples/bug-tests/1489-error-handler-activity-splices.mdl +++ b/mdl-examples/bug-tests/1489-error-handler-activity-splices.mdl @@ -13,8 +13,8 @@ -- -- To reproduce: exec this file, then change `P = $P` to `P = 'x'` (or the -- handler's log text) and exec it again. The second run reports --- `Modified microflow: BugTest1489.Flow (spliced: 1 replaced)`; the handler and --- the merge it rejoins at keep their $IDs and positions. A third run writes +-- `Modified microflow: BugTest1489.Flow (spliced: 1 edited in place)`; the call, +-- the handler and the merge it rejoins at keep their $IDs and positions. A third run writes -- nothing. Changing the handler's kind is still refused. mdl 1; diff --git a/mdl-examples/bug-tests/1517-notes-on-decision-and-end-event-splice.mdl b/mdl-examples/bug-tests/1517-notes-on-decision-and-end-event-splice.mdl index 0e4e40e0a..bb0b0ef43 100644 --- a/mdl-examples/bug-tests/1517-notes-on-decision-and-end-event-splice.mdl +++ b/mdl-examples/bug-tests/1517-notes-on-decision-and-end-event-splice.mdl @@ -11,7 +11,7 @@ -- To reproduce: exec this file, then add `@annotation 'guard'` on the line -- before `if` (or `@annotation 'the end'` before `return 'done';`) and exec it -- again. The second run reports `Modified microflow: BugTest1517.F (spliced: --- 1 replaced)`; the decision and the end event keep their $IDs and flows, and +-- 1 edited in place)`; the decision and the end event keep their $IDs and flows, and -- the note is drawn where a full build would draw it. A third run writes -- nothing. A note shared with another activity (`id:`) is still refused. diff --git a/mdl-examples/bug-tests/1610-decision-caption-change-splices.mdl b/mdl-examples/bug-tests/1610-decision-caption-change-splices.mdl index 86b3f8f82..9bb0254bc 100644 --- a/mdl-examples/bug-tests/1610-decision-caption-change-splices.mdl +++ b/mdl-examples/bug-tests/1610-decision-caption-change-splices.mdl @@ -14,7 +14,7 @@ -- -- To reproduce: exec this file against a project, then add -- `@caption 'Big?'` on the line before `if` and exec it again. The second run --- reports `Modified microflow: BugTest1610.F (spliced: 1 replaced)`, and a +-- reports `Modified microflow: BugTest1610.F (spliced: 1 edited in place)`, and a -- third run writes nothing. Taking the caption off again captions the -- decision with its condition, as a new one would be. diff --git a/mdl-examples/bug-tests/1761-same-kind-activity-edit-keeps-ids.mdl b/mdl-examples/bug-tests/1761-same-kind-activity-edit-keeps-ids.mdl new file mode 100644 index 000000000..735410c30 --- /dev/null +++ b/mdl-examples/bug-tests/1761-same-kind-activity-edit-keeps-ids.mdl @@ -0,0 +1,64 @@ +-- ============================================================================ +-- ako/mxcli#1761 (mendixlabs/mxcli#1667) — an edited filter gets new $IDs +-- ============================================================================ +-- +-- Re-running `create or modify microflow` where the only change is a filter's +-- expression did not edit the activity, it replaced it. Measured after the +-- second run (Mendix 10.24.28): +-- +-- $BigList: the Microflows$ActionActivity, Microflows$ListOperationsAction +-- and Microflows$FilterByExpression all got new $IDs. [...] The decision's +-- ExclusiveSplit and SplitCondition kept theirs. +-- +-- and exec reported both as `spliced: 1 replaced`. Studio Pro compares and +-- merges by $ID, so it showed the filter as one activity removed and another +-- added; check, mx check, diff and describe all looked fine. +-- +-- An activity replaced by one of the same kind — the same action — is now an +-- edit of the stored one: it keeps its $ID, and so do its action and every +-- element under it that pairs up (a member by its attribute or association, +-- an argument by its parameter). Several activities each changed in their own +-- place are each edited so. +-- +-- To reproduce: exec this file against a project, then change every `> 10` +-- to `>= 10`, `= 7` to `= 8` and `Factor = 2` to `Factor = 3`, and exec it +-- again. The second run reports +-- Modified microflow: BugTest1761.FilterSales (spliced: 1 edited in place) +-- Modified microflow: BugTest1761.Decide (spliced: 1 edited in place) +-- Modified microflow: BugTest1761.Kinds (spliced: 3 edited in place) +-- and `mxcli bson dump --type microflow --object BugTest1761.FilterSales` +-- shows the same $IDs as before it; a third run writes nothing. An activity +-- replaced by another kind — +-- a log where the filter was — still reports `replaced with new element IDs`. + +mdl 1; + +create module BugTest1761; + +create or modify persistent entity BugTest1761.Sale ( Amount: Integer, Note: String(200) ); + +create or modify microflow BugTest1761.FilterSales ( $SaleList: List of BugTest1761.Sale ) +returns List of BugTest1761.Sale as $BigList +begin + $SmallList = filter $SaleList where $currentObject/Amount < 5; + $BigList = filter $SaleList where $currentObject/Amount > 10; + return $BigList; +end; + +create or modify microflow BugTest1761.Decide ( $N: Integer ) returns Boolean as $Result +begin + if $N > 10 then return true; else return false; end if; +end; + +create or modify microflow BugTest1761.Sub ( $Sale: BugTest1761.Sale, $Factor: Integer ) returns Boolean as $Ok +begin + return true; +end; + +create or modify microflow BugTest1761.Kinds ( $SaleList: List of BugTest1761.Sale ) returns Boolean as $Ok +begin + $Found = find $SaleList where $currentObject/Amount > 10; + change $Found (Amount = 7, Note = 'x'); + $Ok = call microflow BugTest1761.Sub (Sale = $Found, Factor = 2); + return $Ok; +end; diff --git a/mdl-examples/bug-tests/splice-856-list-operation-where-bare-member.mdl b/mdl-examples/bug-tests/splice-856-list-operation-where-bare-member.mdl index 18ab2f0ec..29ee651ef 100644 --- a/mdl-examples/bug-tests/splice-856-list-operation-where-bare-member.mdl +++ b/mdl-examples/bug-tests/splice-856-list-operation-where-bare-member.mdl @@ -25,10 +25,10 @@ -- -- To reproduce: exec this file against a project. Expected: -- Created microflow: BugTest856.F --- Modified microflow: BugTest856.F (spliced: 1 replaced) +-- Modified microflow: BugTest856.F (spliced: 1 edited in place) -- Before the fix the second statement was refused and exec rolled the whole -- script back. Running the file again sets the declare back and forth, and --- each statement again replaces the declare alone (spliced: 1 replaced). +-- each statement again edits the declare alone (spliced: 1 edited in place). mdl 1; diff --git a/mdl/backend/mfmutator/splice.go b/mdl/backend/mfmutator/splice.go index a1f7a1f6f..e43e7e5c4 100644 --- a/mdl/backend/mfmutator/splice.go +++ b/mdl/backend/mfmutator/splice.go @@ -17,10 +17,14 @@ package mfmutator // every other property of a rewired flow (its $ID, case value, error-handler // flag, the curve at its other end) is kept; // - nodes are moved only to make room, and only by a translation; -// - nothing else is touched, and no $ID is ever rewritten. A removed element -// leaves no reference behind: Save refuses a document in which any binary -// still names an element that was removed (CLAUDE.md rule 1), and one in -// which two elements share an $ID. +// - nothing else is touched, and no stored $ID is ever rewritten. A removed +// element leaves no reference behind: Save refuses a document in which any +// binary still names an element that was removed (CLAUDE.md rule 1), and +// one in which two elements share an $ID; +// - an activity replaced by one of the same kind is edited, not replaced: +// the replacement takes the stored activity's $IDs, and every reference to +// it is rewritten with them in the same pass (splice_edit.go, +// ako/mxcli#1761). // // What the splice cannot do safely it refuses, with the reason: an insert // after a decision (which branch?), before an activity several flows enter @@ -81,6 +85,11 @@ type Mutator struct { // pinnedIn are the same for the nodes of a loop body, by the loop's $ID, // in the loop box's coordinates (PinInLoop). pinnedIn map[string]map[point]bool + // carried maps the $ID a same-kind replacement was built with to the + // stored $ID it takes over, put on at Bytes (ako/mxcli#1761, + // splice_edit.go); edited are the stored activities a Replace edited so. + carried map[string][]byte + edited map[string]bool } var _ backend.MicroflowMutator = (*Mutator)(nil) @@ -96,11 +105,15 @@ func New(doc bson.D, unitID model.ID, deps Deps) (*Mutator, error) { if dDoc(doc, "ObjectCollection") == nil { return nil, fmt.Errorf("unit %s has no object collection", unitID) } - return &Mutator{deps: deps, unitID: unitID, doc: doc, removed: map[string]bool{}}, nil + return &Mutator{deps: deps, unitID: unitID, doc: doc, removed: map[string]bool{}, + carried: map[string][]byte{}, edited: map[string]bool{}}, nil } // Bytes returns the patched unit, after the integrity checks Save applies. +// An activity a replace edited in place takes its stored identity back first +// (applyCarried). func (m *Mutator) Bytes() ([]byte, error) { + m.applyCarried() if err := m.checkIntegrity(); err != nil { return nil, err } @@ -467,6 +480,11 @@ func (m *Mutator) Replace(target model.ID, frag *backend.MicroflowFragment) erro if err != nil { return err } + // One activity of the same kind is an edit of the stored one, which keeps + // its identity (ako/mxcli#1761). + if edited, err := m.editInPlace(g, x, frag); edited || err != nil { + return err + } y := g.nodes[out.dest] if y == nil { return fmt.Errorf("flow %s points at an object that is not in the flow", out.id) @@ -528,7 +546,8 @@ func (m *Mutator) Replace(target model.ID, frag *backend.MicroflowFragment) erro // swapped in place (ako/mxcli#1106): the new activity takes the stored one's // place in the loop's collection, is drawn where it was at the size it was, // and the flows that entered and left it now enter and leave the new one, so -// nothing else in the body moves. By several (ako/mxcli#1693) the fragment +// nothing else in the body moves; of the same kind, it also takes the stored +// one's identity (ako/mxcli#1761). By several (ako/mxcli#1693) the fragment // starts where the activity started, what is past it in the body moves along // when the fragment is longer, and the loop box grows when the body no longer // fits it. The activity may be the body's first (no flow in) or last (no flow @@ -554,7 +573,7 @@ func (m *Mutator) replaceInLoop(g *graph, x *node, frag *backend.MicroflowFragme if d, err := m.serializeInPlace(x, frag); err != nil { return err } else if dString(d, "$Type") == x.typ { - return m.swapInPlace(g, x, dDoc(loop.doc, "ObjectCollection"), d, frag.Entry) + return m.swapKeeping(g, x, dDoc(loop.doc, "ObjectCollection"), d, frag.Entry) } } in := g.incoming(x.id) @@ -828,7 +847,8 @@ func (m *Mutator) dropInLoop(g *graph, x *node) error { // handler by one activity handling errors the same way, in place // (mendixlabs/mxcli#1489): the new activity takes the stored one's place, is // drawn where it was at the size it was, and every flow that entered or left -// it — its error flow included — now enters or leaves the new one. The +// it — its error flow included — now enters or leaves the new one; of the +// same kind, it takes the stored one's identity too (ako/mxcli#1761). The // handler's own activities, and the merge its path rejoins at, are not // touched: changing one of them is a change of its own, addressed by its own // handle. Anything else — more than one activity, or one that does not raise @@ -864,22 +884,12 @@ func (m *Mutator) replaceKeepingHandler(g *graph, x *node, frag *backend.Microfl return fmt.Errorf("cannot replace %s by a %s: it has an error handler, which the splice keeps only on an "+ "activity", describeNode(x), t) } - if err := m.swapInPlace(g, x, dDoc(m.doc, "ObjectCollection"), d, frag.Entry); err != nil { + if err := m.swapKeeping(g, x, dDoc(m.doc, "ObjectCollection"), d, frag.Entry); err != nil { return err } - if notes == nil { - return nil - } - // A note drawn relative to the activity moves with it, from where the - // builder drew the activity to where the stored one is; one whose canvas - // position the script states stays there (mendixlabs/mxcli#1598). - for _, n := range notes.frag.Objects { - if !notes.frag.Pinned[n.GetID()] { - p := n.GetPosition() - n.SetPosition(model.Point{X: p.X + x.pos.X - notes.entryAt.X, Y: p.Y + x.pos.Y - notes.entryAt.Y}) - } - } - return m.addFragment(notes.frag) + // The notes move with the activity, from where the builder drew it to + // where the stored one is. + return m.addNotesAt(notes, x.pos) } // fragmentNotes are the notes a one-activity fragment attaches to its diff --git a/mdl/backend/mfmutator/splice_edit.go b/mdl/backend/mfmutator/splice_edit.go new file mode 100644 index 000000000..45830c419 --- /dev/null +++ b/mdl/backend/mfmutator/splice_edit.go @@ -0,0 +1,332 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mfmutator + +import ( + "bytes" + "fmt" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/model" +) + +// # Editing an activity in place (ako/mxcli#1761) +// +// A replace by one activity of the same kind — the same activity $Type and +// the same action $Type — is an edit of the stored activity, the way Studio +// Pro edits one: the activity keeps its $ID, and so does every element under +// it that the new content has at the same place. Studio Pro compares and +// merges by $ID, so a replace that minted new ones showed an edited filter as +// one activity removed and another added, and turned a teammate's edit of the +// same activity on another branch into a conflict — while a decision whose +// condition changed the same way kept its $IDs (SetCondition). +// +// The replacement is serialized from the statement as for any replace and put +// where the stored activity is (swapInPlace). Its elements are then paired +// with the stored ones structurally (carryPairs), never by a shared value: +// +// - two elements pair when they have the same $Type at the same place: the +// activity itself, or the same key of two paired elements; +// - in a list, entries that name what they are about pair by that name — a +// member change by its attribute or association and whether it sets, adds +// or removes; a parameter mapping by its parameter; a translation by its +// language — so a reordered member keeps its $ID, and a removed member's +// $ID goes onto no other member. A list of unnamed entries pairs by +// position, and only when both lists hold the same $Types in the same +// order; +// - anything else is new: an element whose $Type differs (a find where a +// filter was, a custom range where a constant one was), and everything +// under it. Its stored counterpart is removed, so a reference left to it +// is refused at Save as for any removed element. +// +// The stored $IDs go on at Bytes, not at the replace. Until then the +// replacement carries the $IDs its fragment was built with, which the +// executor still names it by: a stored note linked to it after the replace +// (ako/mxcli#1695), a note the statement states. At Bytes every reference to +// a carried element is rewritten with it, in one pass over every binary in +// the unit, so no pointer is left naming the built $ID (CLAUDE.md rule 1). +// +// What is written is the replacement's content, as before: the statement +// states the activity whole (describe prints every property MDL has a +// spelling for), and the writer completes the property set Studio Pro writes +// (canon.CompletePropertySets). Only which $IDs it carries changes. + +// sameKind reports whether d, a serialized replacement, is the same kind of +// activity as stored: both action activities, whose actions have one $Type. +func sameKind(d, stored bson.D) bool { + const activity = "Microflows$ActionActivity" + if dString(d, "$Type") != activity || dString(stored, "$Type") != activity { + return false + } + na, sa := dDoc(d, "Action"), dDoc(stored, "Action") + return na != nil && sa != nil && dString(na, "$Type") != "" && dString(na, "$Type") == dString(sa, "$Type") +} + +// swapKeeping is swapInPlace that, when d is the same kind of activity as x +// (sameKind), carries x's identity onto it: x's $ID and those of the elements +// under it that pair up (carryPairs), put on at Bytes. +func (m *Mutator) swapKeeping(g *graph, x *node, oc, d bson.D, id model.ID) error { + same := sameKind(d, x.doc) + if err := m.swapInPlace(g, x, oc, d, id); err != nil { + return err + } + if same { + carryPairs(d, x.doc, m.carried) + m.edited[x.id] = true + } + return nil +} + +// EditedInPlace reports whether a Replace of target edited the stored +// activity, keeping its identity, rather than putting new elements in its +// place. +func (m *Mutator) EditedInPlace(target model.ID) bool { return m.edited[string(target)] } + +// editInPlace replaces the top-level activity x by frag's one activity when +// the two are the same kind, and reports whether it did. The replacement is +// written where x is in the collection, drawn where the script states — or, +// when it states nothing, where x is — at x's size (a Studio Pro activity may +// have been resized), and takes x's identity (swapKeeping). The notes frag +// states for the activity come with it, drawn where they are relative to it. +// Anything else — more than one activity, or another kind — is left to the +// replace, with nothing changed. +func (m *Mutator) editInPlace(g *graph, x *node, frag *backend.MicroflowFragment) (bool, error) { + rest, notes := withoutNotes(frag) + if !singleActivity(rest) { + return false, nil + } + obj := rest.Objects[0] + d, err := m.deps.SerializeObject(obj) + if err != nil { + return false, fmt.Errorf("serialize %T: %w", obj, err) + } + if d == nil || !sameKind(d, x.doc) { + return false, nil + } + at := x.pos + if frag.Placed { + p := obj.GetPosition() + at = point{p.X, p.Y} + } + obj.SetPosition(model.Point{X: at.X, Y: at.Y}) + dSet(d, "RelativeMiddlePoint", at.String()) + dSet(d, "Size", x.size.String()) + if err := m.swapKeeping(g, x, dDoc(m.doc, "ObjectCollection"), d, rest.Entry); err != nil { + return true, err + } + return true, m.addNotesAt(notes, at) +} + +// addNotesAt adds the notes a one-activity fragment attaches to its activity, +// now drawn at at: a note drawn relative to the activity moves with it, from +// where the builder drew the activity; one whose canvas position the script +// states stays there (mendixlabs/mxcli#1598). +func (m *Mutator) addNotesAt(notes *fragmentNotes, at point) error { + if notes == nil { + return nil + } + for _, n := range notes.frag.Objects { + if !notes.frag.Pinned[n.GetID()] { + p := n.GetPosition() + n.SetPosition(model.Point{X: p.X + at.X - notes.entryAt.X, Y: p.Y + at.Y - notes.entryAt.Y}) + } + } + return m.addFragment(notes.frag) +} + +// carryPairs records in into, for d and every element under it, the $ID of +// the element of stored it stands for: built $ID (its 16 bytes as a string) → +// stored $ID. d stands for stored when the two have one $Type; see the +// package comment above for how their children and list entries pair. +func carryPairs(d, stored bson.D, into map[string][]byte) { + if t := dString(d, "$Type"); t == "" || t != dString(stored, "$Type") { + return + } + if n, s := elementID(d), elementID(stored); n != nil && s != nil { + into[string(n)] = bytes.Clone(s) + } + for _, e := range d { + if e.Key == "$ID" || e.Key == "$Type" { + continue + } + switch nv := e.Value.(type) { + case bson.D: + if sd, ok := dGet(stored, e.Key).(bson.D); ok { + carryPairs(nv, sd, into) + } + case bson.A: + if sa, ok := dGet(stored, e.Key).(bson.A); ok { + for _, p := range pairEntries(nv, sa) { + carryPairs(p[0], p[1], into) + } + } + } + } +} + +// elementID is d's 16-byte binary $ID, or nil. +func elementID(d bson.D) []byte { + if b, ok := dGet(d, "$ID").(bson.Binary); ok && len(b.Data) == 16 { + return b.Data + } + return nil +} + +// entryNames are the properties by which a list entry names what it is about, +// in the order they are tried: a member change names its attribute or its +// association (one of the two is empty), a parameter mapping its parameter, a +// translation its language. +var entryNames = []string{"Attribute", "Association", "Parameter", "LanguageCode", "Name", "Key"} + +// entryKey is what a list entry is about — its $Type, the first of +// entryNames it states, and its Type where it has one (a member change that +// sets an association is not one that adds to it) — or "" for an entry that +// names nothing. +func entryKey(d bson.D) string { + for _, k := range entryNames { + if s := dString(d, k); s != "" { + key := dString(d, "$Type") + "\x00" + k + "=" + s + if t := dString(d, "Type"); t != "" { + key += "\x00" + t + } + return key + } + } + return "" +} + +// pairEntries pairs the entries of two lists of elements: by entryKey when +// every entry on both sides names what it is about (the n-th entry with a key +// with the n-th stored one with that key), by position when none does and the +// two lists hold the same $Types in the same order, and not at all otherwise. +func pairEntries(news, olds bson.A) [][2]bson.D { + nd, ok1 := entryDocs(news) + od, ok2 := entryDocs(olds) + if !ok1 || !ok2 || len(nd) == 0 || len(od) == 0 { + return nil + } + named, unnamed := 0, 0 + for _, d := range append(append([]bson.D(nil), nd...), od...) { + if entryKey(d) == "" { + unnamed++ + } else { + named++ + } + } + var out [][2]bson.D + switch { + case unnamed == 0: + byKey := map[string][]bson.D{} + for _, d := range od { + k := entryKey(d) + byKey[k] = append(byKey[k], d) + } + for _, d := range nd { + k := entryKey(d) + if q := byKey[k]; len(q) > 0 { + out = append(out, [2]bson.D{d, q[0]}) + byKey[k] = q[1:] + } + } + case named == 0: + if len(nd) != len(od) { + return nil + } + for i := range nd { + if dString(nd[i], "$Type") != dString(od[i], "$Type") { + return nil + } + } + for i := range nd { + out = append(out, [2]bson.D{nd[i], od[i]}) + } + } + return out +} + +// entryDocs returns the elements of a Mendix list, without its marker; false +// when it holds anything but elements (strings, pointers). +func entryDocs(a bson.A) ([]bson.D, bool) { + var out []bson.D + for i, v := range a { + switch t := v.(type) { + case bson.D: + if dString(t, "$Type") == "" { + return nil, false + } + out = append(out, t) + case int32: + if i != 0 { + return nil, false + } + default: + return nil, false + } + } + return out, true +} + +// applyCarried puts on each carried element the stored $ID it takes over, and +// rewrites every reference to it in the same pass: every 16-byte binary in the +// unit that holds the built $ID. A pair applies only while the element is in +// the unit and no element holds the stored $ID — an operation after the +// replace that took the replacement out again leaves the stored $ID removed. +// The stored $IDs that go back on are no longer removed elements. +func (m *Mutator) applyCarried() { + if len(m.carried) == 0 { + return + } + held := map[string]bool{} + collectIDs(m.doc, held) + sub := map[string][]byte{} + taken := map[string]bool{} + for n, s := range m.carried { + if held[n] && !held[string(s)] && !taken[string(s)] { + sub[n] = s + taken[string(s)] = true + } + } + substituteIDs(m.doc, sub) + for _, s := range sub { + delete(m.removed, string(s)) + } + m.carried = map[string][]byte{} +} + +// substituteIDs rewrites, in place, every 16-byte binary in v that sub maps. +func substituteIDs(v any, sub map[string][]byte) { + repl := func(x any) (any, bool) { + b, ok := x.(bson.Binary) + if !ok || len(b.Data) != 16 { + return nil, false + } + if s, ok := sub[string(b.Data)]; ok { + return bson.Binary{Subtype: b.Subtype, Data: bytes.Clone(s)}, true + } + return nil, true + } + switch t := v.(type) { + case bson.D: + for i := range t { + if r, isBin := repl(t[i].Value); isBin { + if r != nil { + t[i].Value = r + } + continue + } + substituteIDs(t[i].Value, sub) + } + case bson.A: + for i := range t { + if r, isBin := repl(t[i]); isBin { + if r != nil { + t[i] = r + } + continue + } + substituteIDs(t[i], sub) + } + } +} diff --git a/mdl/backend/mfmutator/splice_edit_in_place_test.go b/mdl/backend/mfmutator/splice_edit_in_place_test.go new file mode 100644 index 000000000..df1e36492 --- /dev/null +++ b/mdl/backend/mfmutator/splice_edit_in_place_test.go @@ -0,0 +1,441 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mfmutator + +import ( + "bytes" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1761 (mendixlabs/mxcli#1667): re-running `create or modify +// microflow` with only a filter's expression changed did not edit the +// activity, it replaced it — the activity, its action and its operation all +// got new $IDs, while a decision whose condition changed the same way kept its +// own. Studio Pro compares and merges by $ID, so it showed the edited filter +// as one activity removed and another added. +// +// A replace by one activity of the same kind (same activity and action $Type) +// is an edit of the stored activity: the new content is written where the +// stored one is, and the stored $IDs are carried onto every element that pairs +// up — the same key of two paired elements, a list entry with the same name +// (an attribute, an association, a parameter), or the same position in a list +// of unnamed entries of the same length. What does not pair is new. + +// editDeps serializes an activity the way fakeDeps does, with action as its +// Action — the document the codec would write for the fragment's statement. +type editDeps struct { + fakeDeps + action bson.D +} + +func (d *editDeps) SerializeObject(o microflows.MicroflowObject) (bson.D, error) { + doc, err := d.fakeDeps.SerializeObject(o) + if err != nil || d.action == nil { + return doc, err + } + if _, ok := o.(*microflows.ActionActivity); ok && !dSet(doc, "Action", d.action) { + doc = append(doc, bson.E{Key: "Action", Value: d.action}) + } + return doc, nil +} + +// el is an element: its $ID is the named one, or a freshly minted one for "", +// as the builder mints them for a fragment. +func el(name, typ string, props ...bson.E) bson.D { + id := bin(name) + if name == "" { + id = bson.Binary{Data: types.UUIDToBlob(types.GenerateID())} + } + return append(bson.D{{Key: "$ID", Value: id}, {Key: "$Type", Value: typ}}, props...) +} + +func kv(k string, v any) bson.E { return bson.E{Key: k, Value: v} } + +// list is a stored list with its marker. +func list(marker int32, els ...bson.D) bson.A { + out := bson.A{marker} + for _, e := range els { + out = append(out, e) + } + return out +} + +// The six kinds the issue names. Each returns the action with the given $IDs +// ("" mints) and the value under test. +func filterAction(act, op, expr string) bson.D { + return el(act, "Microflows$ListOperationsAction", kv("ErrorHandlingType", "Rollback"), + kv("NewOperation", el(op, "Microflows$FilterByExpression", kv("Expression", expr), kv("ListName", "SaleList"))), + kv("ResultVariableName", "BigList")) +} + +func findAction(act, op, expr string) bson.D { + return el(act, "Microflows$ListOperationsAction", kv("ErrorHandlingType", "Rollback"), + kv("NewOperation", el(op, "Microflows$FindByExpression", kv("Expression", expr), kv("ListName", "SaleList"))), + kv("ResultVariableName", "Found")) +} + +func retrieveAction(act, src, rng, sorts, xpath string) bson.D { + return el(act, "Microflows$RetrieveAction", kv("ErrorHandlingType", "Rollback"), + kv("RetrieveSource", el(src, "Microflows$DatabaseRetrieveSource", kv("Entity", "M.Sale"), + kv("Range", el(rng, "Microflows$ConstantRange", kv("SingleObject", false))), + kv("XpathConstraint", xpath), + kv("NewSortings", el(sorts, "Microflows$SortingsList", kv("Sortings", list(2)))))), + kv("ResultVariableName", "Big")) +} + +func memberItem(id, attr, assoc, value string) bson.D { + return el(id, "Microflows$ChangeActionItem", kv("Attribute", attr), kv("Association", assoc), + kv("Type", "Set"), kv("Value", value)) +} + +func changeAction(typ, act string, items ...bson.D) bson.D { + return el(act, typ, kv("ErrorHandlingType", "Rollback"), kv("Items", list(2, items...)), + kv("RefreshInClient", false), kv("Commit", "No"), kv("ChangeVariableName", "Found")) +} + +func mapping(id, param, arg string) bson.D { + return el(id, "Microflows$MicroflowCallParameterMapping", kv("Argument", arg), kv("Parameter", param)) +} + +func callAction(act, call string, mappings ...bson.D) bson.D { + return el(act, "Microflows$MicroflowCallAction", kv("ErrorHandlingType", "Rollback"), + kv("MicroflowCall", el(call, "Microflows$MicroflowCall", kv("Microflow", "M.SUB"), + kv("ParameterMappings", list(2, mappings...)))), + kv("ResultVariableName", "Ok")) +} + +// withAction is line() with the activity a carrying action. +func withAction(action bson.D) []bson.D { + objs := line() + objs[1] = append(objs[1], bson.E{Key: "Action", Value: action}) + return objs +} + +// replaceA replaces a, which holds stored, by one activity serialized with +// replacement, and returns the mutator, what was saved and the fragment. +func replaceA(t *testing.T, stored, replacement bson.D) (*Mutator, []byte, *backend.MicroflowFragment) { + t.Helper() + raw, err := bson.Marshal(unit(withAction(stored), lineFlows())) + if err != nil { + t.Fatal(err) + } + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatal(err) + } + deps := &editDeps{action: replacement} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + frag := oneActivity() + if err := m.Replace(model.ID(uid("a")), frag); err != nil { + t.Fatalf("replace: %v", err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + return m, deps.saved, frag +} + +func TestSplice_ReplaceBySameKindEditsTheStoredActivity(t *testing.T) { + for _, tc := range []struct { + name string + // stored is a's action as stored; replacement the fragment's, with + // fresh $IDs; want the replacement with the stored $IDs. + stored, replacement, want bson.D + }{ + {"filter expression", + filterAction("act", "op", "$currentObject/Amount > 10"), + filterAction("", "", "$currentObject/Amount >= 10"), + filterAction("act", "op", "$currentObject/Amount >= 10")}, + {"find expression", + findAction("act", "op", "$currentObject/Amount = 3"), + findAction("", "", "$currentObject/Amount = 4"), + findAction("act", "op", "$currentObject/Amount = 4")}, + {"retrieve where", + retrieveAction("act", "src", "rng", "srt", "[Amount > 100]"), + retrieveAction("", "", "", "", "[Amount > 200]"), + retrieveAction("act", "src", "rng", "srt", "[Amount > 200]")}, + {"change member value", + changeAction("Microflows$ChangeAction", "act", memberItem("i1", "M.Sale.Amount", "", "7"), memberItem("i2", "M.Sale.Note", "", "'x'")), + changeAction("Microflows$ChangeAction", "", memberItem("", "M.Sale.Amount", "", "8"), memberItem("", "M.Sale.Note", "", "'x'")), + changeAction("Microflows$ChangeAction", "act", memberItem("i1", "M.Sale.Amount", "", "8"), memberItem("i2", "M.Sale.Note", "", "'x'"))}, + {"create member value", + changeAction("Microflows$CreateChangeAction", "act", memberItem("i1", "M.Sale.Amount", "", "1")), + changeAction("Microflows$CreateChangeAction", "", memberItem("", "M.Sale.Amount", "", "2")), + changeAction("Microflows$CreateChangeAction", "act", memberItem("i1", "M.Sale.Amount", "", "2"))}, + {"call microflow argument", + callAction("act", "call", mapping("p1", "M.SUB.Sale", "$New"), mapping("p2", "M.SUB.Factor", "2")), + callAction("", "", mapping("", "M.SUB.Sale", "$New"), mapping("", "M.SUB.Factor", "3")), + callAction("act", "call", mapping("p1", "M.SUB.Sale", "$New"), mapping("p2", "M.SUB.Factor", "3"))}, + } { + t.Run(tc.name, func(t *testing.T) { + m, saved, _ := replaceA(t, tc.stored, tc.replacement) + // What the edit should leave: the stored unit with a's action + // holding the new value under the stored $IDs — a keeps its + // $ID, its slot in the collection, its place and size, and the + // flows through it are untouched. + want, _ := bson.Marshal(unit(withAction(tc.want), lineFlows())) + if !bytes.Equal(saved, want) { + var got bson.D + _ = bson.Unmarshal(saved, &got) + t.Errorf("the replace did not edit the stored activity in place:\n got %v\nwant %v", got, unit(withAction(tc.want), lineFlows())) + } + if !m.EditedInPlace(model.ID(uid("a"))) { + t.Error("EditedInPlace reports a swapped activity") + } + }) + } +} + +// itemIDs maps each entry of the list at path under a's action to its $ID, by +// the entry's name property. +func itemIDs(t *testing.T, saved []byte, nameKey string, path ...string) map[string]string { + t.Helper() + var d bson.D + if err := bson.Unmarshal(saved, &d); err != nil { + t.Fatal(err) + } + m, _ := New(d, "unit", &fakeDeps{}) + a := m.graph().nodes[uid("a")] + if a == nil { + t.Fatal("a is not in the unit: the activity got a new $ID") + } + cur := dDoc(a.doc, "Action") + for _, k := range path[:len(path)-1] { + cur = dDoc(cur, k) + } + out := map[string]string{} + for _, e := range arrayElements(dGet(cur, path[len(path)-1])) { + if ed, ok := e.(bson.D); ok { + out[dString(ed, nameKey)+dString(ed, "Association")] = binaryID(dGet(ed, "$ID")) + } + } + return out +} + +// A member list pairs by member, never by position: a reordered member keeps +// its own $ID, a removed member's $ID goes with it — onto no other member — +// and an added member is new. +func TestSplice_ReplaceBySameKindPairsMembersByName(t *testing.T) { + t.Run("change members reordered, one dropped, one added", func(t *testing.T) { + _, saved, _ := replaceA(t, + changeAction("Microflows$ChangeAction", "act", + memberItem("i1", "M.Sale.Amount", "", "7"), + memberItem("i2", "M.Sale.Note", "", "'x'"), + memberItem("i3", "", "M.Sale_Customer", "$Cust")), + changeAction("Microflows$ChangeAction", "", + memberItem("", "M.Sale.Note", "", "'y'"), + memberItem("", "M.Sale.Amount", "", "8"), + memberItem("", "M.Sale.Code", "", "'c'"))) + ids := itemIDs(t, saved, "Attribute", "Items") + if ids["M.Sale.Note"] != uid("i2") || ids["M.Sale.Amount"] != uid("i1") { + t.Errorf("a reordered member lost its $ID: %v", ids) + } + if c := ids["M.Sale.Code"]; c == uid("i1") || c == uid("i2") || c == uid("i3") || c == "" { + t.Errorf("the added member took a stored $ID: %s", c) + } + if bytes.Contains(saved, types.UUIDToBlob(uid("i3"))) { + t.Error("the dropped association member's $ID is still in the unit") + } + }) + t.Run("call arguments reordered, one dropped", func(t *testing.T) { + _, saved, _ := replaceA(t, + callAction("act", "call", mapping("p1", "M.SUB.Sale", "$New"), mapping("p2", "M.SUB.Factor", "2")), + callAction("", "", mapping("", "M.SUB.Factor", "3"))) + ids := itemIDs(t, saved, "Parameter", "MicroflowCall", "ParameterMappings") + if len(ids) != 1 || ids["M.SUB.Factor"] != uid("p2") { + t.Errorf("want Factor alone, keeping p2's $ID, got %v", ids) + } + if bytes.Contains(saved, types.UUIDToBlob(uid("p1"))) { + t.Error("the dropped argument's $ID is still in the unit") + } + }) + t.Run("an association set member is not an add member", func(t *testing.T) { + add := memberItem("", "", "M.Sale_Customer", "$Cust") + dSet(add, "Type", "Add") + _, saved, _ := replaceA(t, + changeAction("Microflows$ChangeAction", "act", memberItem("i3", "", "M.Sale_Customer", "$Cust")), + changeAction("Microflows$ChangeAction", "", add)) + if bytes.Contains(saved, types.UUIDToBlob(uid("i3"))) { + t.Error("a member that now adds to the association kept the $ID of the one that set it") + } + }) +} + +// A replace by another kind of activity is not an edit: the activity is new, +// with new $IDs throughout, and the stored one is gone. The same action with +// another operation keeps the activity and the action, and only the operation +// is new. +func TestSplice_ReplaceByAnotherKindIsNew(t *testing.T) { + t.Run("another action", func(t *testing.T) { + m, saved, frag := replaceA(t, + filterAction("act", "op", "$currentObject/Amount > 10"), + changeAction("Microflows$ChangeAction", "", memberItem("", "M.Sale.Amount", "", "8"))) + for _, gone := range []string{"a", "act", "op"} { + if bytes.Contains(saved, types.UUIDToBlob(uid(gone))) { + t.Errorf("%s's $ID is still in the unit", gone) + } + } + if !edges(t, saved)[[2]string{uid("start"), string(frag.Entry)}] { + t.Error("the flow into the replaced activity does not enter the new one") + } + if m.EditedInPlace(model.ID(uid("a"))) { + t.Error("EditedInPlace reports an activity of another kind as edited") + } + }) + t.Run("another operation", func(t *testing.T) { + _, saved, _ := replaceA(t, + filterAction("act", "op", "$currentObject/Amount > 10"), + findAction("", "", "$currentObject/Amount > 10")) + var d bson.D + _ = bson.Unmarshal(saved, &d) + m, _ := New(d, "unit", &fakeDeps{}) + a := m.graph().nodes[uid("a")] + if a == nil { + t.Fatal("the activity got a new $ID") + } + act := dDoc(a.doc, "Action") + if binaryID(dGet(act, "$ID")) != uid("act") { + t.Error("the action got a new $ID") + } + op := dDoc(act, "NewOperation") + if dString(op, "$Type") != "Microflows$FindByExpression" || binaryID(dGet(op, "$ID")) == uid("op") { + t.Errorf("the find kept the filter's $ID: %v", op) + } + if bytes.Contains(saved, types.UUIDToBlob(uid("op"))) { + t.Error("the filter's $ID is still in the unit") + } + }) +} + +// The notes a create or modify states for the activity come with the edit, +// their lines attached to the stored activity; and a stored note linked to the +// fragment's activity after the replace (LinkNote, ako/mxcli#1695) is linked +// to the stored one. +func TestSplice_ReplaceBySameKindKeepsNotesOnTheStoredActivity(t *testing.T) { + objs := append(withAction(filterAction("act", "op", "x > 10")), obj("old", "Microflows$Annotation", 250, 330)) + raw, _ := bson.Marshal(unit(objs, lineFlows())) + var d bson.D + _ = bson.Unmarshal(raw, &d) + deps := &editDeps{action: filterAction("", "", "x >= 10")} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + frag := oneActivity() + note := µflows.Annotation{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Position: model.Point{X: 360, Y: 100}, + }, Caption: "edited"} + frag.Objects = append(frag.Objects, note) + frag.AnnotationFlows = []*microflows.AnnotationFlow{{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + OriginID: note.ID, DestinationID: frag.Entry, + }} + if err := m.Replace(model.ID(uid("a")), frag); err != nil { + t.Fatal(err) + } + if err := m.LinkNote(model.ID(uid("old")), frag.Entry); err != nil { + t.Fatalf("link the stored note to the replacement: %v", err) + } + if err := m.Save(); err != nil { + t.Fatal(err) + } + if bytes.Contains(deps.saved, types.UUIDToBlob(string(frag.Entry))) { + t.Error("the fragment's activity $ID is in the unit: the stored activity was not kept") + } + var got bson.D + _ = bson.Unmarshal(deps.saved, &got) + g, _ := New(got, "unit", &fakeDeps{}) + lines := map[string]bool{} + for _, f := range g.graph().annotationFlows(uid("a")) { + lines[f.origin] = true + } + if !lines[string(note.ID)] || !lines[uid("old")] { + t.Errorf("want the stated note and the stored one attached to a, got %v", lines) + } + if p := g.graph().nodes[string(note.ID)]; p == nil || p.pos != (point{250, 100}) { + t.Errorf("the note is not drawn where it is relative to the activity: %v", p) + } +} + +// Inside a loop body, and on an activity with a custom error handler, the +// replace already swapped one activity in place (ako/mxcli#1106, +// mendixlabs/mxcli#1489); of the same kind, it now keeps the stored identity +// there too. +func TestSplice_ReplaceBySameKindInPlaceKeepsIdentityEverywhere(t *testing.T) { + t.Run("in a loop body", func(t *testing.T) { + inner := append(obj("inner", "Microflows$ActionActivity", 100, 60), bson.E{Key: "Action", Value: filterAction("act", "op", "x > 10")}) + loop := append(obj("loop", "Microflows$LoopedActivity", 420, 200), bson.E{Key: "ObjectCollection", Value: bson.D{ + {Key: "$ID", Value: bin("loopoc")}, + {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "Objects", Value: bson.A{int32(3), inner, obj("inner2", "Microflows$ActionActivity", 260, 60)}}, + }}) + raw, _ := bson.Marshal(unit([]bson.D{obj("start", "Microflows$StartEvent", 100, 200), loop}, + []bson.D{flow("f1", "start", "loop", 1, 3, false), flow("fi", "inner", "inner2", 1, 3, false)})) + var d bson.D + _ = bson.Unmarshal(raw, &d) + deps := &editDeps{action: filterAction("", "", "x >= 10")} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + if err := m.Replace(model.ID(uid("inner")), oneActivity()); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatal(err) + } + for _, kept := range []string{"inner", "act", "op"} { + if !bytes.Contains(deps.saved, types.UUIDToBlob(uid(kept))) { + t.Errorf("%s's $ID is gone", kept) + } + } + if !edges(t, deps.saved)[[2]string{uid("inner"), uid("inner2")}] { + t.Error("the body flow no longer leaves the stored activity") + } + if !m.EditedInPlace(model.ID(uid("inner"))) { + t.Error("EditedInPlace reports a swapped activity") + } + }) + t.Run("with an error handler", func(t *testing.T) { + objs, flows := handlerFlow() + act := callAction("act", "call", mapping("p1", "M.SUB.Factor", "2")) + dSet(act, "ErrorHandlingType", string(microflows.ErrorHandlingTypeCustomWithoutRollback)) + objs[1] = append(objs[1][:len(objs[1])-1], bson.E{Key: "Action", Value: act}) + raw, _ := bson.Marshal(unit(objs, flows)) + var d bson.D + _ = bson.Unmarshal(raw, &d) + repl := callAction("", "", mapping("", "M.SUB.Factor", "3")) + dSet(repl, "ErrorHandlingType", string(microflows.ErrorHandlingTypeCustomWithoutRollback)) + deps := &editDeps{action: repl} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + if err := m.Replace(model.ID(uid("a")), oneActivityEH(microflows.ErrorHandlingTypeCustomWithoutRollback)); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatal(err) + } + for _, kept := range []string{"a", "act", "call", "p1"} { + if !bytes.Contains(deps.saved, types.UUIDToBlob(uid(kept))) { + t.Errorf("%s's $ID is gone", kept) + } + } + e := edges(t, deps.saved) + if !e[[2]string{uid("a"), uid("h")}] || !e[[2]string{uid("start"), uid("a")}] { + t.Errorf("the flows through the activity, its error flow included, do not run through the stored one: %v", e) + } + }) +} diff --git a/mdl/backend/mfmutator/splice_handler_test.go b/mdl/backend/mfmutator/splice_handler_test.go index 5f0df41dd..7e4eb0099 100644 --- a/mdl/backend/mfmutator/splice_handler_test.go +++ b/mdl/backend/mfmutator/splice_handler_test.go @@ -60,35 +60,69 @@ func handlerFlow() ([]bson.D, []bson.D) { } func TestSplice_ReplaceKeepsTheErrorHandler(t *testing.T) { - objs, flows := handlerFlow() - m, deps := newMutator(t, unit(objs, flows)) - frag := oneActivityEH(microflows.ErrorHandlingTypeCustomWithoutRollback) - if err := m.Replace(model.ID(uid("a")), frag); err != nil { - t.Fatal(err) - } - if err := m.Save(); err != nil { - t.Fatalf("save: %v", err) - } - // What the replace should leave: the stored unit with a's document - // swapped for the new one, drawn where a was at a's size, and the three - // flows through a — in, out, and the error flow — pointing at it. - newID := bson.Binary{Data: types.UUIDToBlob(string(frag.Entry))} - wantObjs, wantFlows := handlerFlow() - wantObjs[1] = withEH(bson.D{ - {Key: "$ID", Value: newID}, - {Key: "$Type", Value: "Microflows$ActionActivity"}, - {Key: "RelativeMiddlePoint", Value: "250;200"}, - {Key: "Size", Value: "160;60"}, - }, microflows.ErrorHandlingTypeCustomWithoutRollback) - dSet(wantFlows[0], "DestinationPointer", newID) - dSet(wantFlows[1], "OriginPointer", newID) - dSet(wantFlows[2], "OriginPointer", newID) - raw, _ := bson.Marshal(unit(wantObjs, wantFlows)) - if !bytes.Equal(deps.saved, raw) { - var got bson.D - _ = bson.Unmarshal(deps.saved, &got) - t.Errorf("the replace changed more than the activity and the flows through it:\n got %v\nwant %v", got, unit(wantObjs, wantFlows)) - } + // The same kind of activity (a call for a call) is an edit of the stored + // one, which keeps its $ID (ako/mxcli#1761): the replacement equal to it, + // the unit is the stored one, byte for byte. + t.Run("the same kind of activity", func(t *testing.T) { + objs, flows := handlerFlow() + m, deps := newMutator(t, unit(objs, flows)) + if err := m.Replace(model.ID(uid("a")), oneActivityEH(microflows.ErrorHandlingTypeCustomWithoutRollback)); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + raw, _ := bson.Marshal(unit(handlerFlow())) + if !bytes.Equal(deps.saved, raw) { + var got bson.D + _ = bson.Unmarshal(deps.saved, &got) + t.Errorf("the replace by the same activity changed the unit:\n got %v\nwant %v", got, unit(handlerFlow())) + } + }) + // Another kind is a new activity: what the replace should leave is the + // stored unit with a's document swapped for the new one, drawn where a + // was at a's size, and the three flows through a — in, out, and the + // error flow — pointing at it. + t.Run("another kind of activity", func(t *testing.T) { + objs, flows := handlerFlow() + raw, _ := bson.Marshal(unit(objs, flows)) + var d bson.D + _ = bson.Unmarshal(raw, &d) + java := bson.D{ + {Key: "$Type", Value: "Microflows$JavaActionCallAction"}, + {Key: "ErrorHandlingType", Value: string(microflows.ErrorHandlingTypeCustomWithoutRollback)}, + } + deps := &editDeps{action: java} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + frag := oneActivityEH(microflows.ErrorHandlingTypeCustomWithoutRollback) + if err := m.Replace(model.ID(uid("a")), frag); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + newID := bson.Binary{Data: types.UUIDToBlob(string(frag.Entry))} + wantObjs, wantFlows := handlerFlow() + wantObjs[1] = bson.D{ + {Key: "$ID", Value: newID}, + {Key: "$Type", Value: "Microflows$ActionActivity"}, + {Key: "RelativeMiddlePoint", Value: "250;200"}, + {Key: "Size", Value: "160;60"}, + {Key: "Action", Value: java}, + } + dSet(wantFlows[0], "DestinationPointer", newID) + dSet(wantFlows[1], "OriginPointer", newID) + dSet(wantFlows[2], "OriginPointer", newID) + want, _ := bson.Marshal(unit(wantObjs, wantFlows)) + if !bytes.Equal(deps.saved, want) { + var got bson.D + _ = bson.Unmarshal(deps.saved, &got) + t.Errorf("the replace changed more than the activity and the flows through it:\n got %v\nwant %v", got, unit(wantObjs, wantFlows)) + } + }) } func TestSplice_ReplaceOfAHandledActivityRefusals(t *testing.T) { diff --git a/mdl/backend/microflow_mutation.go b/mdl/backend/microflow_mutation.go index 8b8b4f085..9ae785682 100644 --- a/mdl/backend/microflow_mutation.go +++ b/mdl/backend/microflow_mutation.go @@ -47,8 +47,16 @@ type MicroflowMutator interface { InsertAfter(target model.ID, frag *MicroflowFragment) error // InsertBefore splices frag onto the one flow that enters target. InsertBefore(target model.ID, frag *MicroflowFragment) error - // Replace puts frag in target's place and removes target. + // Replace puts frag in target's place and removes target. A target + // replaced by one activity of the same kind — the same activity and + // action — is edited instead: the new content is written onto the stored + // activity, which keeps its $ID and those of the elements under it that + // pair up (ako/mxcli#1761). Replace(target model.ID, frag *MicroflowFragment) error + // EditedInPlace reports whether a Replace of target edited the stored + // activity, keeping its identity, rather than putting new elements in + // its place — what the report of the change says. + EditedInPlace(target model.ID) bool // Drop removes target and joins its incoming flows to its successor. A // decision of the shape `alter … insert` produces goes with its paths and // its merge (ako/mxcli#1694); any other decision is refused. diff --git a/mdl/executor/cmd_alter_flow.go b/mdl/executor/cmd_alter_flow.go index 06ed6c4de..30d6c192e 100644 --- a/mdl/executor/cmd_alter_flow.go +++ b/mdl/executor/cmd_alter_flow.go @@ -151,6 +151,7 @@ func (a *alterFlowContext) applyTo(ctx *ExecContext, mut backend.MicroflowMutato if err := a.setNodeNotes(ctx, mut, op, target); err != nil { return fail(err) } + a.noteEdited(op) continue } if op.SetCondition { @@ -164,6 +165,7 @@ func (a *alterFlowContext) applyTo(ctx *ExecContext, mut backend.MicroflowMutato return fail(err) } a.noteRewritten(target, "if "+expr) + a.noteEdited(op) continue } if ret, ok := returnValueEdit(op, target); ok { @@ -177,6 +179,7 @@ func (a *alterFlowContext) applyTo(ctx *ExecContext, mut backend.MicroflowMutato return fail(err) } a.noteRewritten(target, "return "+value) + a.noteEdited(op) continue } frag, vars, err := a.buildFragmentVars(ctx, op.Body) @@ -234,7 +237,11 @@ func (a *alterFlowContext) applyTo(ctx *ExecContext, mut backend.MicroflowMutato return nil }) } - err = mut.Replace(target.ID, frag) + if err = mut.Replace(target.ID, frag); err == nil && mut.EditedInPlace(target.ID) { + // One activity of the same kind: the stored one, edited + // (ako/mxcli#1761). + a.noteEdited(op) + } default: err = fmt.Errorf("unknown operation") } @@ -309,6 +316,21 @@ type alterFlowContext struct { // each (ako/mxcli#1695). nil until then — and for the patch of a `create or // modify`, which states its notes and has none to resolve. storedNotes map[string]*microflows.Annotation + + // edited are the replace operations that edited the stored node in place + // — a decision's condition, a return's value, a node's notes, an activity + // replaced by one of the same kind — so that it kept its $ID, as against + // one whose target was swapped for new elements (ako/mxcli#1761). The + // report of a `create or modify` tells the two apart. + edited map[*ast.AlterFlowOperation]bool +} + +// noteEdited records that op edited its stored target in place. +func (a *alterFlowContext) noteEdited(op *ast.AlterFlowOperation) { + if a.edited == nil { + a.edited = map[*ast.AlterFlowOperation]bool{} + } + a.edited[op] = true } // candidatesOf returns the stored candidates with the given $IDs. diff --git a/mdl/executor/cmd_flow_loop_body_edit_pedapp_test.go b/mdl/executor/cmd_flow_loop_body_edit_pedapp_test.go index 856646225..4f1e2c358 100644 --- a/mdl/executor/cmd_flow_loop_body_edit_pedapp_test.go +++ b/mdl/executor/cmd_flow_loop_body_edit_pedapp_test.go @@ -114,7 +114,7 @@ func TestCreateOrModify_EditInsideLoopBody(t *testing.T) { if err := agreeExec(t, exec, edited); err != nil { t.Fatalf("an expression edit inside a loop body is refused: %v", err) } - if !strings.Contains(out.String(), "Modified microflow: MyFirstModule.LoopEdit (spliced: 1 replaced)") { + if !strings.Contains(out.String(), "Modified microflow: MyFirstModule.LoopEdit (spliced: 1 edited in place)") { t.Errorf("want the edit reported as one splice, got:\n%s", out.String()) } desc := describeFlow(t, exec, "LoopEdit") @@ -122,8 +122,9 @@ func TestCreateOrModify_EditInsideLoopBody(t *testing.T) { t.Errorf("the edit was not written:\n%s", desc) } - // Everything but the replaced activity keeps its $ID and its place: the - // loop, its other activity, and the flow around it. + // Every object keeps its $ID and its place: the loop, its other + // activity, and the edited activity itself, which is the stored one + // with a new expression (ako/mxcli#1761) — the same kind of activity. after := flowObjectIDs(loopEditFlowOf(t, exec, "LoopEdit").ObjectCollection) var gone, added []model.ID for id := range before { @@ -138,11 +139,8 @@ func TestCreateOrModify_EditInsideLoopBody(t *testing.T) { t.Errorf("object %s moved from %v to %v", id, q, p) } } - if len(gone) != 1 || len(added) != 1 { - t.Fatalf("want exactly the edited activity replaced, got %d gone, %d added", len(gone), len(added)) - } - if before[gone[0]] != after[added[0]] { - t.Errorf("the replacement is drawn at %v, the activity it replaces at %v", after[added[0]], before[gone[0]]) + if len(gone) != 0 || len(added) != 0 { + t.Fatalf("want the edited activity kept, got %d gone, %d added", len(gone), len(added)) } // The re-run is a fixed point. @@ -200,8 +198,8 @@ func TestCreateOrModify_EditInsideLoopBody(t *testing.T) { if err := agreeExec(t, exec, src); err != nil { t.Fatalf("refused: %v", err) } - if !strings.Contains(out.String(), "2 replaced") { - t.Errorf("want two in-place replaces, got:\n%s", out.String()) + if !strings.Contains(out.String(), "2 edited in place") { + t.Errorf("want two in-place edits, got:\n%s", out.String()) } desc := describeFlow(t, exec, "LoopEdit") if !strings.Contains(desc, "'still in loop'") || !strings.Contains(desc, "'start'") { diff --git a/mdl/executor/cmd_flow_modify.go b/mdl/executor/cmd_flow_modify.go index d6ceb4329..5e93c59f4 100644 --- a/mdl/executor/cmd_flow_modify.go +++ b/mdl/executor/cmd_flow_modify.go @@ -187,7 +187,7 @@ func modifyFlowInPlace(ctx *ExecContext, d *flowDecl) (handled bool, err error) containerID = to } - switch summary := patchSummary(ops, moves, set); { + switch summary := patchSummary(ops, a.edited, moves, set); { case summary != "": ctx.ReportMutation("Modified", "%s: %s (%s)", d.kind(), d.name, summary) case moved: @@ -1029,6 +1029,17 @@ func (pd *patchDiff) gap(ins []ast.MicroflowStatement, stored []ast.MicroflowSta } } } + // A run of activities each changed in its own place is that many changes + // of one activity, so each can be edited in place (ako/mxcli#1761) rather + // than the run being replaced as a whole. + if pairwiseSameKind(ins, del) { + for i := range ins { + if err := pd.gap(ins[i:i+1], stored, si+i, si+i+1); err != nil { + return err + } + } + return nil + } cands := make([]mfmutator.Candidate, len(del)) for i, st := range del { c, err := pd.loc.locate(st) @@ -1453,6 +1464,16 @@ func (pd *patchDiff) bodyGap(lc mfmutator.Candidate, at ast.MicroflowStatement, return nil } } + // A run of activities each changed in its own place: one at a time, as + // at top level (ako/mxcli#1761). + if pairwiseSameKind(ins, del) { + for i := range ins { + if err := pd.bodyGap(lc, at, ins[i:i+1], stored, si+i, si+i+1); err != nil { + return err + } + } + return nil + } if i, j, ok := loopPair(ins, del); ok { if err := pd.bodyGap(lc, at, ins[:i], stored, si, si+j); err != nil { return err @@ -1612,6 +1633,24 @@ func replaceableInPlace(declared, stored ast.MicroflowStatement) bool { return true } +// pairwiseSameKind reports whether a run of more than one changed statement +// is each declared statement standing where the stored one of the same kind +// stood: as many of each, every pair the same kind of plain activity (not a +// decision, a loop, a return, a break or continue, nor one with an error +// handler). Each pair is then a change of one activity, which the splice +// edits in place when the activity stays the same kind (ako/mxcli#1761). +func pairwiseSameKind(ins, del []ast.MicroflowStatement) bool { + if len(ins) < 2 || len(ins) != len(del) { + return false + } + for i := range ins { + if statementKind(ins[i]) != statementKind(del[i]) || notPlainInBody(ins[i]) != "" || notPlainInBody(del[i]) != "" { + return false + } + } + return true +} + // customErrorHandler reports whether a statement has an `on error` with a // body: a handler path of its own, drawn in the body beside it. func customErrorHandler(st ast.MicroflowStatement) bool { @@ -1988,15 +2027,23 @@ func targetLabel(c mfmutator.Candidate) string { } // patchSummary says what a patch did, e.g. "spliced: 1 inserted, 2 moved; -// set: ExportLevel", or "" when it did nothing. -func patchSummary(ops []*ast.AlterFlowOperation, moves []flowMove, set []string) string { - var ins, rep, drop int +// set: ExportLevel", or "" when it did nothing. A replace that edited its +// stored node in place (edited) — which keeps its $ID, and so reads in Studio +// Pro's version control as that node changed — is told apart from one that +// put new elements in its place, which reads as one removed and another added +// (ako/mxcli#1761). +func patchSummary(ops []*ast.AlterFlowOperation, edited map[*ast.AlterFlowOperation]bool, moves []flowMove, set []string) string { + var ins, edit, rep, drop int for _, op := range ops { switch op.Op { case ast.AlterFlowInsertAfter, ast.AlterFlowInsertBefore: ins++ case ast.AlterFlowReplace, ast.AlterFlowReplaceBody: - rep++ + if edited[op] { + edit++ + } else { + rep++ + } case ast.AlterFlowDrop: drop++ } @@ -2005,7 +2052,7 @@ func patchSummary(ops []*ast.AlterFlowOperation, moves []flowMove, set []string) for _, p := range []struct { n int verb string - }{{ins, "inserted"}, {rep, "replaced"}, {drop, "dropped"}, {len(moves), "moved"}} { + }{{ins, "inserted"}, {edit, "edited in place"}, {rep, "replaced with new element IDs"}, {drop, "dropped"}, {len(moves), "moved"}} { if p.n > 0 { parts = append(parts, fmt.Sprintf("%d %s", p.n, p.verb)) } diff --git a/mdl/executor/flow_splice_decision_1694_pedapp_test.go b/mdl/executor/flow_splice_decision_1694_pedapp_test.go index e14fae397..a44db8580 100644 --- a/mdl/executor/flow_splice_decision_1694_pedapp_test.go +++ b/mdl/executor/flow_splice_decision_1694_pedapp_test.go @@ -96,10 +96,12 @@ func TestSpliceDecision1694_ReRunTakesTheInsertedDecisionOut(t *testing.T) { }) } - // The control for the $ID assertion: an activity the script does change - // shows up as one $ID gone and one added, so the comparison sees churn - // where there is some. (A rebuild is no control here: a flow dropped and - // created again in one session gets its stored identities back.) + // The control for the $ID assertion: an activity the script replaces by + // another kind shows up as one $ID gone and one added, so the comparison + // sees churn where there is some. (A rebuild is no control here: a flow + // dropped and created again in one session gets its stored identities + // back; nor is a changed log text, which edits the stored activity in + // place and keeps its $ID, ako/mxcli#1761.) t.Run("control: a replaced activity is churn", func(t *testing.T) { const name = "D1694Control" original := fmt.Sprintf(dec1694Flow, name) @@ -107,7 +109,7 @@ func TestSpliceDecision1694_ReRunTakesTheInsertedDecisionOut(t *testing.T) { t.Fatal(err) } before := flowObjectIDs(loopEditFlowOf(t, exec, name).ObjectCollection) - if err := agreeExec(t, exec, strings.Replace(original, "'start'", "'begin'", 1)); err != nil { + if err := agreeExec(t, exec, strings.Replace(original, "log info node 'X' 'start';", "commit $F;", 1)); err != nil { t.Fatal(err) } if missing, extra := sameIDs(before, flowObjectIDs(loopEditFlowOf(t, exec, name).ObjectCollection)); missing != 1 || extra != 1 { diff --git a/mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go b/mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go new file mode 100644 index 000000000..e0973e3ce --- /dev/null +++ b/mdl/executor/flow_splice_edit_in_place_1761_pedapp_test.go @@ -0,0 +1,364 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "fmt" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// ako/mxcli#1761 (mendixlabs/mxcli#1667, reported on Mendix 10.24.28): re-run +// `create or modify microflow` with only a filter's expression changed, and +// mxcli did not edit the activity, it replaced it: +// +// $BigList: the Microflows$ActionActivity, Microflows$ListOperationsAction +// and Microflows$FilterByExpression all got new $IDs. [...] The decision's +// ExclusiveSplit and SplitCondition kept theirs. +// +// and exec reported both as `spliced: 1 replaced`. Studio Pro compares and +// merges by $ID, so it showed the filter as one activity removed and another +// added. An activity replaced by one of the same kind is now an edit of the +// stored one, which keeps its $ID and those of its action and the elements +// under it; exec says which it was. + +const edit1761Setup = `mdl 1; +create persistent entity MyFirstModule.Sale1761 ( Amount: Integer, Note: String(200) ); +create persistent entity MyFirstModule.Customer1761 ( Name: String(200) ); +create association MyFirstModule.Sale1761_Customer1761 from MyFirstModule.Sale1761 to MyFirstModule.Customer1761; +create microflow MyFirstModule.SUB1761 ( $Sale: MyFirstModule.Sale1761, $Factor: Integer ) returns Boolean as $Ok +begin + return true; +end; +create nanoflow MyFirstModule.NSUB1761 ( $Sale: MyFirstModule.Sale1761, $Factor: Integer ) returns Boolean as $Ok +begin + return true; +end;` + +// edit1761Flow is the flow under test, a microflow or a nanoflow (kind), which +// calls callee with call ("call microflow", "call nanoflow"). +func edit1761Flow(kind, call, callee string) string { + return fmt.Sprintf(`mdl 1; +create or modify %s MyFirstModule.Edit1761 ( $SaleList: List of MyFirstModule.Sale1761, $Cust: MyFirstModule.Customer1761 ) +returns Boolean as $Ok +begin + $Small = filter $SaleList where $currentObject/Amount < 5; + $Found = find $SaleList where $currentObject/Amount = 3; + retrieve $Big from MyFirstModule.Sale1761 where Amount > 100; + change $Found (Amount = 7, Note = 'x', MyFirstModule.Sale1761_Customer1761 = $Cust); + $New = create MyFirstModule.Sale1761 (Amount = 1, Note = 'n'); + $Ok = %s MyFirstModule.%s (Sale = $New, Factor = 2); + return $Ok; +end;`, kind, call, callee) +} + +// storedFlowUnit returns the stored bytes of the microflow or nanoflow +// module.name. +func storedFlowUnit(t *testing.T, exec *Executor, module, name string) []byte { + t.Helper() + ctx := exec.newExecContext(t.Context()) + h, err := getHierarchy(ctx) + if err != nil { + t.Fatal(err) + } + mfs, err := ctx.Backend.ListMicroflows() + if err != nil { + t.Fatal(err) + } + for _, m := range mfs { + if m.Name == name && h.GetModuleName(h.FindModuleID(m.ContainerID)) == module { + raw, err := ctx.Backend.GetRawUnitBytes(m.ID) + if err != nil { + t.Fatal(err) + } + return append([]byte(nil), raw...) + } + } + nfs, err := ctx.Backend.ListNanoflows() + if err != nil { + t.Fatal(err) + } + for _, n := range nfs { + if n.Name == name && h.GetModuleName(h.FindModuleID(n.ContainerID)) == module { + raw, err := ctx.Backend.GetRawUnitBytes(n.ID) + if err != nil { + t.Fatal(err) + } + return append([]byte(nil), raw...) + } + } + t.Fatalf("%s.%s not found", module, name) + return nil +} + +// storedActivity is one activity of a stored flow: its $ID, the $ID of every +// element under it by path, and how many sequence flows enter and leave it. +type storedActivity struct { + id string + ids map[string]string + in, out int + contents []byte +} + +// findActivity returns the top-level activity whose action is an actType that +// outputs v. +func findActivity(t *testing.T, raw []byte, actType, v string) storedActivity { + t.Helper() + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatal(err) + } + oc, _ := afGet(d, "ObjectCollection").(bson.D) + var found bson.D + for _, el := range afList(afGet(oc, "Objects")) { + o, _ := el.(bson.D) + a, _ := afGet(o, "Action").(bson.D) + if a == nil || afGet(a, "$Type") != actType { + continue + } + for _, k := range []string{"ResultVariableName", "VariableName", "ChangeVariableName", "OutputVariableName"} { + if s, _ := afGet(a, k).(string); s == v { + found = o + } + } + } + if found == nil { + t.Fatalf("no %s outputting $%s in the stored flow", actType, v) + } + sa := storedActivity{id: afIDOf(afGet(found, "$ID")), ids: map[string]string{}} + sa.contents, _ = bson.Marshal(found) + var walk func(v any, path string) + walk = func(v any, path string) { + switch n := v.(type) { + case bson.D: + if id, ok := afGet(n, "$ID").(bson.Binary); ok { + sa.ids[path] = types.BlobToUUID(id.Data) + } + for _, e := range n { + walk(e.Value, path+"/"+e.Key) + } + case bson.A: + for i, el := range n { + name := fmt.Sprint(i) + if ed, ok := el.(bson.D); ok { + for _, k := range []string{"Attribute", "Association", "Parameter"} { + if s, _ := afGet(ed, k).(string); s != "" { + name = s + } + } + } + walk(el, path+"["+name+"]") + } + } + } + walk(found, "") + for _, el := range afList(afGet(d, "Flows")) { + f, _ := el.(bson.D) + if afGet(f, "$Type") != "Microflows$SequenceFlow" { + continue + } + if afIDOf(afGet(f, "OriginPointer")) == sa.id { + sa.out++ + } + if afIDOf(afGet(f, "DestinationPointer")) == sa.id { + sa.in++ + } + } + return sa +} + +// assertSameIdentity fails unless after is before with every element $ID +// kept and its flows still running through it. +func assertSameIdentity(t *testing.T, before, after storedActivity) { + t.Helper() + if after.id != before.id { + t.Errorf("the activity got a new $ID: %s -> %s", before.id, after.id) + } + for path, id := range before.ids { + if got := after.ids[path]; got != id { + t.Errorf("%s: $ID %s -> %s", path, id, got) + } + } + for path := range after.ids { + if _, ok := before.ids[path]; !ok { + t.Errorf("%s: an element that was not there", path) + } + } + if after.in != before.in || after.out != before.out { + t.Errorf("flows through the activity: %d in, %d out, want %d in, %d out", after.in, after.out, before.in, before.out) + } +} + +func TestCreateOrModify_SameKindActivityIsEditedInPlace(t *testing.T) { + for _, kind := range []struct{ flow, call, callee string }{ + {"microflow", "call microflow", "SUB1761"}, + {"nanoflow", "call nanoflow", "NSUB1761"}, + } { + base := edit1761Flow(kind.flow, kind.call, kind.callee) + for _, tc := range []struct { + name, from, to string + actType, v string + want string // in the stored activity after the edit + }{ + {"filter expression", "$currentObject/Amount < 5", "$currentObject/Amount <= 5", + "Microflows$ListOperationsAction", "Small", "$currentObject/Amount <= 5"}, + {"find expression", "$currentObject/Amount = 3", "$currentObject/Amount = 4", + "Microflows$ListOperationsAction", "Found", "$currentObject/Amount = 4"}, + {"retrieve where", "where Amount > 100", "where Amount > 200", + "Microflows$RetrieveAction", "Big", "[Amount > 200]"}, + {"change member value", "Amount = 7", "Amount = 8", + "Microflows$ChangeAction", "Found", "8"}, + {"create member value", "Amount = 1", "Amount = 2", + "Microflows$CreateChangeAction", "New", "2"}, + {"call argument", "Factor = 2", "Factor = 3", + "", "Ok", "3"}, + } { + t.Run(kind.flow+"/"+tc.name, func(t *testing.T) { + actType := tc.actType + if actType == "" { + actType = "Microflows$MicroflowCallAction" + if kind.flow == "nanoflow" { + actType = "Microflows$NanoflowCallAction" + } + } + exec, out, _ := openPedAppCopy(t) + if err := agreeExec(t, exec, edit1761Setup); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + if err := agreeExec(t, exec, base); err != nil { + t.Fatalf("create: %v\n%s", err, out.String()) + } + before := findActivity(t, storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"), actType, tc.v) + + edited := strings.Replace(base, tc.from, tc.to, 1) + if edited == base { + t.Fatalf("%q is not in the script", tc.from) + } + out.Reset() + if err := agreeExec(t, exec, edited); err != nil { + t.Fatalf("the edit is refused: %v\n%s", err, out.String()) + } + if want := "(spliced: 1 edited in place)"; !strings.Contains(out.String(), want) { + t.Errorf("want %q, got:\n%s", want, out.String()) + } + after := findActivity(t, storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"), actType, tc.v) + assertSameIdentity(t, before, after) + if !bytes.Contains(after.contents, []byte(tc.want)) { + t.Errorf("the new value %q was not written", tc.want) + } + // The edit is the whole of the change: running it again + // writes nothing. + out.Reset() + if err := agreeExec(t, exec, edited); err != nil { + t.Fatal(err) + } + if !strings.Contains(out.String(), "Unchanged") { + t.Errorf("re-running the edited script wrote again:\n%s", out.String()) + } + }) + } + } +} + +// Control: an activity replaced by another kind of activity is new — new +// $IDs, and the report says so. +func TestCreateOrModify_AnotherKindOfActivityIsReplacedWithNewIDs(t *testing.T) { + exec, out, _ := openPedAppCopy(t) + if err := agreeExec(t, exec, edit1761Setup); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + base := edit1761Flow("microflow", "call microflow", "SUB1761") + if err := agreeExec(t, exec, base); err != nil { + t.Fatalf("create: %v\n%s", err, out.String()) + } + before := findActivity(t, storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"), "Microflows$ListOperationsAction", "Small") + + // A find in place of the filter: the same action, another operation — + // the activity and its action are the stored ones, the operation is new. + out.Reset() + if err := agreeExec(t, exec, strings.Replace(base, "$Small = filter", "$Small = find", 1)); err != nil { + t.Fatalf("%v\n%s", err, out.String()) + } + after := findActivity(t, storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"), "Microflows$ListOperationsAction", "Small") + if after.id != before.id || after.ids["/Action"] != before.ids["/Action"] { + t.Errorf("the activity or its action got a new $ID") + } + if after.ids["/Action/NewOperation"] == before.ids["/Action/NewOperation"] { + t.Errorf("the find kept the filter's $ID") + } + + // A log in place of the filter is another activity. + out.Reset() + if err := agreeExec(t, exec, strings.Replace(base, "$Small = filter $SaleList where $currentObject/Amount < 5;", + "log info node 'T' 'no filter';", 1)); err != nil { + t.Fatalf("%v\n%s", err, out.String()) + } + if want := "(spliced: 1 replaced with new element IDs)"; !strings.Contains(out.String(), want) { + t.Errorf("want %q, got:\n%s", want, out.String()) + } + if raw := storedFlowUnit(t, exec, "MyFirstModule", "Edit1761"); bytes.Contains(raw, types.UUIDToBlob(before.id)) { + t.Error("the filter's $ID is still in the flow") + } +} + +// The subject that can fail: a flow Studio Pro drew (PedApp's +// FeedbackModule.PopulateUserAttributes), whose retrieve and change share one +// note. Its describe output, with the retrieve's where and one change member +// edited, keeps both activities, their actions and every element under them, +// and the note stays on both. +func TestCreateOrModify_StudioProActivitiesAreEditedInPlace(t *testing.T) { + exec, out, _ := openPedAppCopy(t) + var buf bytes.Buffer + saved := exec.output + exec.output = &buf + if err := afRun(t, exec, "describe microflow FeedbackModule.PopulateUserAttributes;"); err != nil { + t.Fatal(err) + } + exec.output = saved + desc := buf.String() + desc = desc[:strings.Index(desc, "\nend;")+len("\nend;")] + + raw := storedFlowUnit(t, exec, "FeedbackModule", "PopulateUserAttributes") + retrieve := findActivity(t, raw, "Microflows$RetrieveAction", "CurrentUser") + change := findActivity(t, raw, "Microflows$ChangeAction", "Feedback") + notes := func(raw []byte) map[string]int { + var d bson.D + _ = bson.Unmarshal(raw, &d) + n := map[string]int{} + for _, el := range afList(afGet(d, "Flows")) { + f, _ := el.(bson.D) + if afGet(f, "$Type") == "Microflows$AnnotationFlow" { + n[afIDOf(afGet(f, "OriginPointer"))+">"+afIDOf(afGet(f, "DestinationPointer"))]++ + } + } + return n + } + notesBefore := notes(raw) + + edited := strings.Replace(desc, "where id = $currentUser", "where Name = $currentUser/Name", 1) + edited = strings.Replace(edited, "SubmitterDisplayName = $CurrentUser/Name", "SubmitterDisplayName = trim($CurrentUser/Name)", 1) + if edited == desc || strings.Count(edited, "trim(") != 1 || !strings.Contains(edited, "where Name = $currentUser/Name") { + t.Fatalf("the describe output does not hold the statements under test:\n%s", desc) + } + out.Reset() + if err := agreeExec(t, exec, edited); err != nil { + t.Fatalf("%v\n%s", err, out.String()) + } + if want := "(spliced: 2 edited in place)"; !strings.Contains(out.String(), want) { + t.Errorf("want %q, got:\n%s", want, out.String()) + } + raw = storedFlowUnit(t, exec, "FeedbackModule", "PopulateUserAttributes") + assertSameIdentity(t, retrieve, findActivity(t, raw, "Microflows$RetrieveAction", "CurrentUser")) + assertSameIdentity(t, change, findActivity(t, raw, "Microflows$ChangeAction", "Feedback")) + if got := notes(raw); fmt.Sprint(got) != fmt.Sprint(notesBefore) { + t.Errorf("the note's lines changed: %v -> %v", notesBefore, got) + } + if !bytes.Contains(raw, []byte("trim($CurrentUser/Name)")) || !bytes.Contains(raw, []byte("[Name = $currentUser/Name]")) { + t.Error("the edits were not written") + } +} diff --git a/mdl/executor/flow_splice_error_handler_pedapp_test.go b/mdl/executor/flow_splice_error_handler_pedapp_test.go index 794668a5a..2e0a9423e 100644 --- a/mdl/executor/flow_splice_error_handler_pedapp_test.go +++ b/mdl/executor/flow_splice_error_handler_pedapp_test.go @@ -70,9 +70,11 @@ func handlerObjects(t *testing.T, exec *Executor, name string) (map[model.ID]mod } // spliceStep runs src — check first, then exec — and asserts it is spliced -// with exactly `changed` objects replaced and `added` new ones, nothing else -// rebuilt or moved, that describe shows want, and that a second run is -// Unchanged. +// with exactly `changed` objects replaced by new ones and `added` new ones, +// nothing else rebuilt or moved, that describe shows want, and that a second +// run is Unchanged. An activity changed into the same kind of activity is +// edited in place and keeps its $ID (ako/mxcli#1761), so it is not among the +// `changed`: describe showing want is what says it was written. func spliceStep(t *testing.T, exec *Executor, out *bytes.Buffer, name, src string, changed, added int, want ...string) { t.Helper() before, _ := handlerObjects(t, exec, name) @@ -133,19 +135,21 @@ func TestCreateOrModify_ActivityWithErrorHandlerIsSplicedInPlace(t *testing.T) { return "mdl 1;\n" + src[strings.Index(src, "create or modify microflow MyFirstModule.HFlow"):] } + // The call is edited in place (ako/mxcli#1761): it keeps its $ID, and + // describe shows the new argument. t.Run("T1 call argument changed, handler unchanged", func(t *testing.T) { _, call := handlerObjects(t, exec, "HFlow") - spliceStep(t, exec, out, "HFlow", flow(strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1)), 1, 0, + spliceStep(t, exec, out, "HFlow", flow(strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1)), 0, 0, "HSub(P = 'x') on error without rollback begin", "log warning node 'Repro' 'failed';") - if _, now := handlerObjects(t, exec, "HFlow"); now == call { - t.Error("the call kept its $ID: the change was not written") + if _, now := handlerObjects(t, exec, "HFlow"); now != call { + t.Error("the call got a new $ID for a changed argument") } }) t.Run("T3 only the handler's log changed", func(t *testing.T) { _, call := handlerObjects(t, exec, "HFlow") src := strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1) src = strings.Replace(src, "'failed';", "'failed again';", 1) - spliceStep(t, exec, out, "HFlow", flow(src), 1, 0, "'failed again'", "HSub(P = 'x') on error") + spliceStep(t, exec, out, "HFlow", flow(src), 0, 0, "'failed again'", "HSub(P = 'x') on error") if _, now := handlerObjects(t, exec, "HFlow"); now != call { t.Error("the call was replaced for a change inside its handler") } @@ -153,7 +157,7 @@ func TestCreateOrModify_ActivityWithErrorHandlerIsSplicedInPlace(t *testing.T) { t.Run("call and handler changed together", func(t *testing.T) { src := strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'y') on", 1) src = strings.Replace(src, "'failed';", "'failed twice';", 1) - spliceStep(t, exec, out, "HFlow", flow(src), 2, 0, "HSub(P = 'y') on error", "'failed twice'") + spliceStep(t, exec, out, "HFlow", flow(src), 0, 0, "HSub(P = 'y') on error", "'failed twice'") }) // A note on the activity is part of it: added or reworded with the // activity unchanged otherwise, it is written, not dropped with the @@ -162,21 +166,21 @@ func TestCreateOrModify_ActivityWithErrorHandlerIsSplicedInPlace(t *testing.T) { src := strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'y') on", 1) src = strings.Replace(src, "'failed';", "'failed twice';", 1) src = strings.Replace(src, " call microflow MyFirstModule.HSub(", " @annotation 'calls the sub'\n call microflow MyFirstModule.HSub(", 1) - spliceStep(t, exec, out, "HFlow", flow(src), 1, 1, "@annotation 'calls the sub'", "HSub(P = 'y') on error") + spliceStep(t, exec, out, "HFlow", flow(src), 0, 1, "@annotation 'calls the sub'", "HSub(P = 'y') on error") }) t.Run("note on the call reworded", func(t *testing.T) { src := strings.Replace(handlerFlowSrc, "HSub(P = $P) on", "HSub(P = 'y') on", 1) src = strings.Replace(src, "'failed';", "'failed twice';", 1) src = strings.Replace(src, " call microflow MyFirstModule.HSub(", " @annotation 'calls the sub, once'\n call microflow MyFirstModule.HSub(", 1) - // The stored note goes out with the activity, the reworded one comes - // in: two objects replaced. - spliceStep(t, exec, out, "HFlow", flow(src), 2, 0, "@annotation 'calls the sub, once'") + // The stored note goes out, the reworded one comes in: one object + // replaced. The call itself is the stored one. + spliceStep(t, exec, out, "HFlow", flow(src), 1, 0, "@annotation 'calls the sub, once'") if desc := describeFlow(t, exec, "HFlow"); strings.Contains(desc, "'calls the sub'") { t.Errorf("the old note is still there:\n%s", desc) } }) t.Run("T4 handler that returns", func(t *testing.T) { - spliceStep(t, exec, out, "HRet", strings.Replace(handlerRetSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1), 1, 0, + spliceStep(t, exec, out, "HRet", strings.Replace(handlerRetSrc, "HSub(P = $P) on", "HSub(P = 'x') on", 1), 0, 0, "HSub(P = 'x') on error without rollback begin") }) @@ -248,9 +252,17 @@ func TestAlter_ReplaceOfActivityWithErrorHandler(t *testing.T) { if err := agreeExec(t, exec, src); err != nil { t.Fatalf("refused: %v", err) } + // The same kind of activity: the stored call, edited in place + // (ako/mxcli#1761) — every object keeps its $ID, and describe shows + // the new argument. after, now := handlerObjects(t, exec, "HFlow") - if now == call || len(after) != len(before) { - t.Errorf("want the call alone replaced: %d -> %d objects", len(before), len(after)) + if now != call || len(after) != len(before) { + t.Errorf("want the call edited in place: call %s -> %s, %d -> %d objects", call, now, len(before), len(after)) + } + for id := range before { + if _, ok := after[id]; !ok { + t.Errorf("object %s got a new $ID", id) + } } for id, p := range before { if q, ok := after[id]; ok && p != q { diff --git a/mdl/executor/flow_splice_list_op_where_856_pedapp_test.go b/mdl/executor/flow_splice_list_op_where_856_pedapp_test.go index 9bbbd7ac4..ca39b19fb 100644 --- a/mdl/executor/flow_splice_list_op_where_856_pedapp_test.go +++ b/mdl/executor/flow_splice_list_op_where_856_pedapp_test.go @@ -58,15 +58,16 @@ func TestCreateOrModify_ListOperationWhereBareMemberIsItsStoredActivity(t *testi if strings.Contains(out.String(), "MDL-V1-REBUILD") { t.Errorf("the change fell back to rebuilding the flow:\n%s", out.String()) } - if !strings.Contains(out.String(), "Modified microflow: MyFirstModule.ListOpWhere (spliced: 1 replaced)") { - t.Errorf("want only the changed declare replaced, got:\n%s", out.String()) + if !strings.Contains(out.String(), "Modified microflow: MyFirstModule.ListOpWhere (spliced: 1 edited in place)") { + t.Errorf("want only the changed declare edited, got:\n%s", out.String()) } if got := describeFlow(t, exec, "ListOpWhere"); !strings.Contains(got, "declare $N Integer = 1;") { t.Errorf("the change was not written:\n%s", got) } - // Only the declare is new: the find, the filter and the rest keep - // their $IDs and places. + // Nothing is new: the find, the filter and the rest keep their + // $IDs and places, and so does the declare, edited in place + // (ako/mxcli#1761). after := flowObjectIDs(loopEditFlowOf(t, exec, "ListOpWhere").ObjectCollection) var gone, added []model.ID for id := range before { @@ -81,8 +82,8 @@ func TestCreateOrModify_ListOperationWhereBareMemberIsItsStoredActivity(t *testi t.Errorf("object %s moved from %v to %v", id, q, p) } } - if len(gone) != 1 || len(added) != 1 { - t.Errorf("want exactly the declare replaced (1 gone, 1 new), got %d gone, %d new", len(gone), len(added)) + if len(gone) != 0 || len(added) != 0 { + t.Errorf("want every object kept (0 gone, 0 new), got %d gone, %d new", len(gone), len(added)) } }) } @@ -103,8 +104,8 @@ func TestCreateOrModify_ListOperationWhereChangedPredicateIsAChange(t *testing.T if err := agreeExec(t, exec, strings.Replace(listOpWhereFlow, c.from, c.to, 1)); err != nil { t.Fatalf("%s → %s: %v", c.from, c.to, err) } - if !strings.Contains(out.String(), "(spliced: 1 replaced)") { - t.Errorf("%s → %s: want the operation replaced, got:\n%s", c.from, c.to, out.String()) + if !strings.Contains(out.String(), "(spliced: 1 edited in place)") { + t.Errorf("%s → %s: want the operation edited, got:\n%s", c.from, c.to, out.String()) } // Back to the stored statement for the next case. if err := agreeExec(t, exec, listOpWhereFlow); err != nil { diff --git a/mdl/executor/flow_splice_loop_body_1693_pedapp_test.go b/mdl/executor/flow_splice_loop_body_1693_pedapp_test.go index a319980c8..1a1899fc9 100644 --- a/mdl/executor/flow_splice_loop_body_1693_pedapp_test.go +++ b/mdl/executor/flow_splice_loop_body_1693_pedapp_test.go @@ -215,7 +215,7 @@ func TestSpliceLoopBody1693_CreateOrModify(t *testing.T) { {"LB1693Swap", " log info node 'X' 'middle';\n set $S = $S + 'a';\n commit $It;\n", "", []string{"'middle'", "set $S", "commit $It"}, 1, 1}, {"LB1693ReplaceByTwo", " set $S = $S + 'a';\n log info node 'X' 'one';\n log info node 'X' 'two';\n commit $It;\n", - "spliced: 1 replaced", []string{"set $S", "'one'", "'two'", "commit $It"}, 1, 2}, + "spliced: 1 replaced with new element IDs", []string{"set $S", "'one'", "'two'", "commit $It"}, 1, 2}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { diff --git a/mdl/roundtrip/flow_commit_events_rerun_test.go b/mdl/roundtrip/flow_commit_events_rerun_test.go index 20ea9eb6c..1dbc59840 100644 --- a/mdl/roundtrip/flow_commit_events_rerun_test.go +++ b/mdl/roundtrip/flow_commit_events_rerun_test.go @@ -64,8 +64,8 @@ func TestSpliceRerun_CommitWithEventsIsTheStoredCommit(t *testing.T) { if err := h.exec(edited); err != nil { t.Fatalf("change: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { - t.Errorf("change: want one statement replaced:\n%s", h.out.String()) + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { + t.Errorf("change: want one statement edited in place:\n%s", h.out.String()) } if len(before.diff(h.snapshot())) == 0 { t.Fatal("change: the real change wrote nothing") diff --git a/mdl/roundtrip/flow_idempotent_shapes_test.go b/mdl/roundtrip/flow_idempotent_shapes_test.go index da0f7e27d..0b98011bc 100644 --- a/mdl/roundtrip/flow_idempotent_shapes_test.go +++ b/mdl/roundtrip/flow_idempotent_shapes_test.go @@ -191,7 +191,7 @@ end; want string }{ // The final return: the stored else of the described form. - {"a changed final return", []edit{{" return $N;\nend;", " return $N + $Base;\nend;"}}, "spliced: 1 replaced"}, + {"a changed final return", []edit{{" return $N;\nend;", " return $N + $Base;\nend;"}}, "spliced: 1 edited in place"}, // An activity inserted between the guard and the final return. {"an activity after the guard", []edit{{" return $N;\nend;", " declare $M Integer = $N * 2;\n return $M;\nend;"}}, "spliced: 1 inserted"}, // Two changes straddling the guard's return: one each, never one @@ -199,7 +199,7 @@ end; {"changes on both sides of the guard's return", []edit{ {"declare $Base Integer = 1;", "declare $Base Integer = 2;"}, {"declare $Neg Integer = 0 - $N;", "declare $Neg Integer = 1 - $N;"}, - }, "spliced: 2 replaced"}, + }, "spliced: 2 edited in place"}, } { t.Run(c.name, func(t *testing.T) { h.restore() diff --git a/mdl/roundtrip/flow_modify_end_test.go b/mdl/roundtrip/flow_modify_end_test.go index e21931902..205013eea 100644 --- a/mdl/roundtrip/flow_modify_end_test.go +++ b/mdl/roundtrip/flow_modify_end_test.go @@ -100,7 +100,7 @@ func TestFlowModify_ReturnValueIsSetInPlace(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("want a splice report, got:\n%s", h.out.String()) } after := h.flowUnit(t, "ConvertUUIDToURL") @@ -140,7 +140,7 @@ func TestFlowModify_InsertBeforeChangedReturn(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 inserted, 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 inserted, 1 edited in place)") { t.Errorf("want a splice report, got:\n%s", h.out.String()) } requireKept(t, before, h.flowUnit(t, "ConvertUUIDToURL"), "") diff --git a/mdl/roundtrip/flow_modify_notes_test.go b/mdl/roundtrip/flow_modify_notes_test.go index 629499f6f..465963fff 100644 --- a/mdl/roundtrip/flow_modify_notes_test.go +++ b/mdl/roundtrip/flow_modify_notes_test.go @@ -197,8 +197,8 @@ end; t.Fatalf("exec 1: %v\n%s", err, h.out.String()) } for _, want := range []string{ - "Modified microflow: MyFirstModule.Rerun_Rollback (spliced: 1 replaced)", - "Modified nanoflow: MyFirstModule.Rerun_RollbackNf (spliced: 1 replaced)", + "Modified microflow: MyFirstModule.Rerun_Rollback (spliced: 1 edited in place)", + "Modified nanoflow: MyFirstModule.Rerun_RollbackNf (spliced: 1 edited in place)", } { // The control is in the same line: the change itself is written. if !strings.Contains(h.out.String(), want) { diff --git a/mdl/roundtrip/flow_modify_test.go b/mdl/roundtrip/flow_modify_test.go index 2b5e75556..5025faf8c 100644 --- a/mdl/roundtrip/flow_modify_test.go +++ b/mdl/roundtrip/flow_modify_test.go @@ -86,8 +86,8 @@ func TestFlowModify_MDL1ReadsStoredStringsAsStored(t *testing.T) { if bytes.Equal(h.flowUnit(t, "VAL_Feedback"), before) { t.Fatal("mdl 1: a string that states another value than the stored one wrote nothing") } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { - t.Errorf("mdl 1: want the one statement replaced by a splice, got:\n%s", h.out.String()) + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { + t.Errorf("mdl 1: want the one statement edited in place by a splice, got:\n%s", h.out.String()) } if again := h.mustDescribeMdl0(t, valFeedback); !strings.Contains(again, "characters\\\\n';") { t.Errorf("mdl 1: want the backslash stored as written:\n%s", again) @@ -199,7 +199,7 @@ func TestFlowModify_DropAndReplace(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec edited definition: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced, 1 dropped)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place, 1 dropped)") { t.Errorf("want a splice report, got:\n%s", h.out.String()) } after := h.flowUnit(t, "SUB_Feedback_Sanitize") @@ -231,7 +231,7 @@ func TestFlowModify_BranchEditIsSpliced(t *testing.T) { if err := h.exec(edited); err != nil { t.Fatalf("exec edited definition: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("want a splice report, got:\n%s", h.out.String()) } after := h.flowUnit(t, "VAL_Feedback") @@ -334,7 +334,7 @@ end;` if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("a value edit in the loop body under mdl 1: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("a value edit in the loop body: want a splice, got:\n%s", h.out.String()) } described = h.mustDescribeMdl0(t, target) @@ -371,7 +371,7 @@ end;` if err := h.exec("mdl 1;\n" + after); err != nil { t.Fatalf("control under mdl 1: %v", err) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("control: want a splice, got:\n%s", h.out.String()) } } diff --git a/mdl/roundtrip/flow_respelling_rerun_test.go b/mdl/roundtrip/flow_respelling_rerun_test.go index 677b60d7c..19ce7fd37 100644 --- a/mdl/roundtrip/flow_respelling_rerun_test.go +++ b/mdl/roundtrip/flow_respelling_rerun_test.go @@ -188,7 +188,9 @@ func respelledTwice(t *testing.T, h *harness, label, header, flow, setup, script } // Control: a real change outside any loop body is written, and - // moves nothing it does not change. + // moves nothing it does not change. The changed statement is the + // same kind of activity, so it is edited in place and keeps its $ID + // too (ako/mxcli#1761). edited := strings.Replace(script, from, to, 1) if edited == script { t.Fatalf("%s: the script has no %q", label, from) @@ -198,8 +200,8 @@ func respelledTwice(t *testing.T, h *harness, label, header, flow, setup, script if err := h.exec(header + edited); err != nil { t.Fatalf("%s: control: %v\n%s", label, err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { - t.Errorf("%s: control: want one statement replaced:\n%s", label, h.out.String()) + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { + t.Errorf("%s: control: want one statement edited in place:\n%s", label, h.out.String()) } if len(before.diff(h.snapshot())) == 0 { t.Fatalf("%s: control: the real change wrote nothing", label) @@ -209,19 +211,19 @@ func respelledTwice(t *testing.T, h *harness, label, header, flow, setup, script for id, p := range stored { q, ok := after[id] if !ok { - continue // the replaced activity + continue } kept++ if q != p { t.Errorf("%s: control: the unchanged %s moved from %s to %s", label, id, p, q) } } - if kept != len(stored)-1 || len(after) != len(stored) { - t.Errorf("%s: control: %d of %d stored objects kept, %d after; want all but the replaced one", + if kept != len(stored) || len(after) != len(stored) { + t.Errorf("%s: control: %d of %d stored objects kept, %d after; want every one, the edited activity too", label, kept, len(stored), len(after)) } if !samePositions(stored, after) { - t.Errorf("%s: control: the replacement is not drawn where the replaced activity was", label) + t.Errorf("%s: control: the edited activity is not drawn where it was", label) } // The twice-exec rule for the change. second := h.snapshot() diff --git a/mdl/roundtrip/flow_splice_guard_test.go b/mdl/roundtrip/flow_splice_guard_test.go index 3dce8eb4c..4ff29ee47 100644 --- a/mdl/roundtrip/flow_splice_guard_test.go +++ b/mdl/roundtrip/flow_splice_guard_test.go @@ -249,7 +249,7 @@ func TestSpliceRerun_StudioProDecisionConditionSetInPlace(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 edited in place)") { t.Errorf("want the condition set, got:\n%s", h.out.String()) } after := h.flowUnit(t, "ChangePassword") @@ -356,7 +356,7 @@ func TestSpliceRerun_ReplaceEndingInGuard(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 replaced with new element IDs)") { t.Errorf("want one replace, got:\n%s", h.out.String()) } after := h.flowUnit(t, "ShowPasswordForm") @@ -397,7 +397,7 @@ end; if err := h.exec("mdl 1;\n" + guarded); err != nil { t.Fatalf("guard: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "(spliced: 1 replaced)") { + if !strings.Contains(h.out.String(), "(spliced: 1 replaced with new element IDs)") { t.Errorf("want one replace, got:\n%s", h.out.String()) } requireDecisionCases(t, h.flowUnit(t, "Grow_Last")) diff --git a/mdl/roundtrip/flow_splice_handler_return_test.go b/mdl/roundtrip/flow_splice_handler_return_test.go index 8a090b57e..95672d719 100644 --- a/mdl/roundtrip/flow_splice_handler_return_test.go +++ b/mdl/roundtrip/flow_splice_handler_return_test.go @@ -54,7 +54,7 @@ begin end error; log info node 'EH' 'done'; end; -`, "(spliced: 1 replaced)", 2}, +`, "(spliced: 1 replaced with new element IDs)", 2}, {"insert, mdl 1, a returned value", "mdl 1;\n", `create or modify microflow MyFirstModule.EH_Insert1 () returns Boolean as $R begin diff --git a/mdl/roundtrip/flow_splice_member_qualified_test.go b/mdl/roundtrip/flow_splice_member_qualified_test.go index bae027fb4..9325bfda9 100644 --- a/mdl/roundtrip/flow_splice_member_qualified_test.go +++ b/mdl/roundtrip/flow_splice_member_qualified_test.go @@ -95,8 +95,8 @@ func TestFlowModify_SplicedMembersAreQualified(t *testing.T) { case err != nil: t.Errorf("%s: exec: %v", label, err) continue - case !strings.Contains(h.out.String(), "(spliced: 1 replaced)"): - t.Errorf("%s: want one statement replaced by a splice, got:\n%s", label, h.out.String()) + case !strings.Contains(h.out.String(), "(spliced: 1 edited in place)"): + t.Errorf("%s: want one statement edited in place by a splice, got:\n%s", label, h.out.String()) } spliced := flowMembers(t, h.flowUnit(t, "T885_Flow")) for _, m := range spliced { diff --git a/mdl/roundtrip/flow_splice_parity_test.go b/mdl/roundtrip/flow_splice_parity_test.go index 10674fb7a..cb0ffa381 100644 --- a/mdl/roundtrip/flow_splice_parity_test.go +++ b/mdl/roundtrip/flow_splice_parity_test.go @@ -137,8 +137,8 @@ func TestPedAppSpliceParity_AuthoredSpellings(t *testing.T) { if err := h.exec("mdl 1;\n" + edited); err != nil { t.Fatalf("exec an edited constraint: %v\n%s", err, h.out.String()) } - if !strings.Contains(h.out.String(), "Modified microflow: MyFirstModule.SpliceParity (spliced: 1 replaced)") { - t.Errorf("an edited constraint must be spliced in as one replaced retrieve:\n%s", h.out.String()) + if !strings.Contains(h.out.String(), "Modified microflow: MyFirstModule.SpliceParity (spliced: 1 edited in place)") { + t.Errorf("an edited constraint must be spliced in as one retrieve edited in place:\n%s", h.out.String()) } if len(created.diff(h.snapshot())) == 0 { t.Error("an edited constraint wrote nothing") diff --git a/mdl/roundtrip/rerun_idempotent_test.go b/mdl/roundtrip/rerun_idempotent_test.go index 5f8d921bf..7246c3705 100644 --- a/mdl/roundtrip/rerun_idempotent_test.go +++ b/mdl/roundtrip/rerun_idempotent_test.go @@ -441,7 +441,7 @@ end; } written := len(before.diff(h.snapshot())) switch { - case run == 1 && (written == 0 || !strings.Contains(h.out.String(), "(spliced: 1 replaced)")): + case run == 1 && (written == 0 || !strings.Contains(h.out.String(), "(spliced: 1 edited in place)")): t.Errorf("run 1: want the edited set spliced in, %d unit(s) written:\n%s", written, h.out.String()) case run == 2 && written != 0: t.Errorf("run 2 of the same script wrote %d unit(s):\n%s", written, h.out.String()) From 76efadf3e9209edfdb856cfc9408af37ad308edc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 23:16:35 +0000 Subject: [PATCH 22/44] docs(microflows): say what a same-kind replace keeps The alter syntax help, the alter-microflow reference page and the microflow and nanoflow skills now say that an activity replaced by one of the same kind is the stored activity edited, keeping its $ID and those of its action and the parts that pair up, and how exec reports it (`edited in place` against `replaced with new element IDs`) (ako/mxcli#1761). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .../skills/mendix/write-microflows/SKILL.md | 2 +- .../write-microflows/reference/pitfalls.md | 5 +-- .../skills/mendix/write-nanoflows/SKILL.md | 34 ++++++++++--------- cmd/mxcli/syntax/features_microflow.go | 6 ++++ .../reference/microflow/alter-microflow.md | 21 ++++++++++++ 5 files changed, 49 insertions(+), 19 deletions(-) diff --git a/.claude/skills/mendix/write-microflows/SKILL.md b/.claude/skills/mendix/write-microflows/SKILL.md index 8e3fccb1e..9cc999c03 100644 --- a/.claude/skills/mendix/write-microflows/SKILL.md +++ b/.claude/skills/mendix/write-microflows/SKILL.md @@ -69,7 +69,7 @@ Choose the mode by who owns the microflow ([choose-edit-mode](../choose-edit-mod (targets from `describe microflow X with handles`). `create or modify` of `describe` output patches too: unchanged writes nothing; a statement (even one whose handler returns), guard clause, `return` value, `if` condition, header clause, parameter (added/retyped; removed only if unused) or stated `@position`/`@start` change is - patched in place (a move keeps the node's flows), as are plain activities added, removed, reordered or changed in a loop body ([an insert needing room past a stated `@position`](reference/pitfalls.md#13-inserting-where-the-script-states-every-position)). A redrawn + patched in place (a move keeps the node's flows; an activity changed into one of the same kind — an expression, a `where`, a member value, an argument — keeps its `$ID` and its action's, reported `edited in place`), as are plain activities added, removed, reordered or changed in a loop body ([an insert needing room past a stated `@position`](reference/pitfalls.md#13-inserting-where-the-script-states-every-position)). A redrawn `@anchor`/`@curve`, a decision/loop/`return` added or changed inside a loop body, error handler or other `return` added/taken away rebuilds under mdl 0 (`MDL-V1-REBUILD`: IDs renumbered, merges and curves lost) and is refused under `mdl 1;`. **To put a decision or loop in a loop body, `alter … replace body of loop $X in $Xs with begin … end;`** — the loop keeps its `$ID` ([pitfalls](reference/pitfalls.md#11-changing-something-inside-a-loop-body)); a decision `alter … insert` put in is taken out by a re-run, `drop` or `replace` ([pitfalls](reference/pitfalls.md#15-taking-out-a-decision-another-script-put-in)). **To rebuild deliberately, `drop` and `create` in ONE script** — the grants carry only within it ([pitfalls](reference/pitfalls.md#drop--create-is-still-a-new-document)). diff --git a/.claude/skills/mendix/write-microflows/reference/pitfalls.md b/.claude/skills/mendix/write-microflows/reference/pitfalls.md index 3d812a9a3..8f1cd8342 100644 --- a/.claude/skills/mendix/write-microflows/reference/pitfalls.md +++ b/.claude/skills/mendix/write-microflows/reference/pitfalls.md @@ -305,8 +305,9 @@ inserted, dropped, moved (dropped and inserted again) or replaced by one or more others (ako/mxcli#1693), and one replaced by one of the same kind in place (ako/mxcli#1106). The new activities are laid out among the body's nodes; when the body no longer fits, the loop box grows on its far side and what is drawn past the -loop moves along. Only the changed activities get new `$ID`s — the loop, the rest of -its body and everything outside it keep theirs. The loop's iterator is in scope, and +loop moves along. Only the new activities get new `$ID`s — one replaced by the same +kind of activity is the stored one edited and keeps its `$ID` and its action's +(ako/mxcli#1761); the loop, the rest of its body and everything outside it keep theirs. The loop's iterator is in scope, and a nested loop's body is edited the same way. `create or modify` does this for a body whose statements change; `alter` for an activity of the body addressed by its handle. diff --git a/.claude/skills/mendix/write-nanoflows/SKILL.md b/.claude/skills/mendix/write-nanoflows/SKILL.md index 6740be6af..e035dd784 100644 --- a/.claude/skills/mendix/write-nanoflows/SKILL.md +++ b/.claude/skills/mendix/write-nanoflows/SKILL.md @@ -11,21 +11,21 @@ description: "Nanoflow syntax in MDL — shared with microflows, but client-side | lines | section | |---|---| | 35-44 | When to Use This Skill | -| 45-81 | Changing an Existing Nanoflow | -| 82-100 | When to Use a Nanoflow vs a Microflow | -| 101-115 | Key Differences from Microflows | -| 116-139 | Nanoflow Structure | -| 140-148 | Naming Convention | -| 149-217 | Supported Activities | -| 218-484 | Real-World Patterns | -| 485-555 | NanoflowCommons JavaScript Actions Reference | -| 556-594 | Disallowed Activities | -| 595-598 | Return Type Restrictions | -| 599-643 | Error Handling in Nanoflows | -| 644-651 | Security (GRANT/REVOKE) | -| 652-664 | Management Commands | -| 665-675 | Common Mistakes | -| 676-690 | Validation Checklist | +| 45-83 | Changing an Existing Nanoflow | +| 84-102 | When to Use a Nanoflow vs a Microflow | +| 103-117 | Key Differences from Microflows | +| 118-141 | Nanoflow Structure | +| 142-150 | Naming Convention | +| 151-219 | Supported Activities | +| 220-486 | Real-World Patterns | +| 487-557 | NanoflowCommons JavaScript Actions Reference | +| 558-596 | Disallowed Activities | +| 597-600 | Return Type Restrictions | +| 601-645 | Error Handling in Nanoflows | +| 646-653 | Security (GRANT/REVOKE) | +| 654-666 | Management Commands | +| 667-677 | Common Mistakes | +| 678-692 | Validation Checklist | # Mendix Nanoflow Skill @@ -52,7 +52,9 @@ Choose the mode by who owns the nanoflow ([choose-edit-mode](../choose-edit-mode (targets by output variable, caption or statement pattern). `create or modify` of `describe` output also works as a patch: an unchanged definition writes nothing, and an inserted, replaced or dropped statement (at the top level or in an `if` branch) is spliced in, - leaving every other node, merge and curve as stored — a guard clause (`if … then return; + leaving every other node, merge and curve as stored — a statement changed into one of + the same kind (an expression, a `where`, a member value, an argument) edits the stored + activity, which keeps its `$ID` and its action's (reported `edited in place`); a guard clause (`if … then return; end if;`) too, its return a new end event, and so is the `return` ending a new activity's error handler; a changed `return` value or `if` condition is set on the stored end event or decision, and a body without a trailing `return` means the stored end. The diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 1f12a16bf..9cfd46949 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -624,6 +624,12 @@ func init() { "-- stored flow before any operation runs; an ambiguous or unknown target is an error.\n" + "-- Only the new activities, the rewired flows and the objects moved to make room change;\n" + "-- every other element keeps its $ID, position and curve.\n" + + "-- REPLACE by one activity of the same kind (the same action: a filter's expression, a\n" + + "-- retrieve's where, a member's value, a call's argument changed) edits the stored\n" + + "-- activity: it keeps its $ID and those of its action and the parts that pair up — a\n" + + "-- member by its attribute or association, an argument by its parameter (ako/mxcli#1761).\n" + + "-- CREATE OR MODIFY reports it as `edited in place`; anything else replaced gets new\n" + + "-- element IDs, reported as `replaced with new element IDs`.\n" + "-- Inside a loop body, plain activities are inserted, replaced and dropped the same way,\n" + "-- the loop box growing when the body no longer fits (ako/mxcli#1693).\n" + "-- REPLACE BODY OF keeps the loop itself (its $ID, position, iterator and list)\n" + diff --git a/docs-site/src/reference/microflow/alter-microflow.md b/docs-site/src/reference/microflow/alter-microflow.md index 69536e27f..1f3685207 100644 --- a/docs-site/src/reference/microflow/alter-microflow.md +++ b/docs-site/src/reference/microflow/alter-microflow.md @@ -32,6 +32,20 @@ against the flow as stored before any operation runs. A fragment is written exactly as the same statements are in `CREATE MICROFLOW`, between `BEGIN` and `END`. +### Editing an activity + +`REPLACE` by one activity of the same kind — the same action, with a +filter's or find's expression, a retrieve's `WHERE`, a member's value or a +call's argument changed — edits the stored activity rather than putting a new +one in its place. The activity keeps its element ID, its place and its size, +and so do its action and every element under it that pairs up: a member of a +change or create by its attribute or association, an argument by its +parameter, anything else by its property and type. A member taken out goes +with its ID, which no other member inherits; a part of another type (a `FIND` +where a `FILTER` was) is new. Studio Pro, which compares and merges by element +ID, then shows the activity as changed rather than one removed and another +added. An activity replaced by another kind of activity, or by several, is new. + ### Inside a loop body Inside a loop body, plain activities are inserted, replaced and dropped the @@ -84,6 +98,13 @@ makes it — for a decision inside a loop body, `REPLACE BODY OF` that loop. Re-running the script that created a flow after another script spliced a decision into it takes that decision out again, when it has the shape above. +A statement changed where it stands edits its stored activity, as above, and +so does each of several adjacent statements changed in their own places. The +report says which: `Modified microflow: M.F (spliced: 1 edited in place)` for +a node that kept its element ID — an activity of the same kind, a decision's +condition, a return's value — and `replaced with new element IDs` for one +that did not. + ## Examples Insert, replace and drop at top level: From 35741a2c69eb816b26407d8e0d66e6a06c4fd6ad Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 22:38:44 +0000 Subject: [PATCH 23/44] fix(canon): leave the end-of-path marker out of a parallel split below Mendix 10.21 Symptom (ako/mxcli#1163): on Mendix 9.x and 10.0-10.20 a workflow with a `parallel split` made the project unloadable. exec reported "Created workflow", then mx check died with "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$EndOfParallelSplitPathActivity". Cause: the builder and the ALTER mutator end every path with that marker, because from 10.21 the engine runs a path only up to it. The type itself was introduced in 10.21.0 (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared a written $Type with the project's version. Older shape, measured: Studio Pro 9.4.0 (the Marketplace's "Example project workflow extension" 2.0.0) stores a split whose ParallelSplitOutcome is {$ID,$Type,Flow} and whose paths hold their activities and nothing more. Lowering produces exactly that key set. Fix: canon.LowerToProjectShape drops the marker from every Workflows$Flow below 10.21, for every writer at once (create, create or modify, and a path alter workflow inserts). Measured on fresh apps, the bug-test exec'd then mx check: - 9.24.30.51827: before TypeCacheUnknownTypeException, after 0 errors; mx convert 9.24.30 changes no key; describe -> exec "Unchanged". - 10.12.19.81612: before TypeCacheUnknownTypeException, after 0 errors; mx convert 10.12.19 changes no key. - 10.24.24 and 11.13.0: 4 markers, 0 errors, key sets identical to the unfixed build. Control: stubbing the lowering fails TestNormalize_ParallelSplitPathsBelowMendix10_21 on 9.24 and 10.20 with the marker still in each path. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-28-parallel-split-marker-below-10-21.json | 1 + CHANGELOG.md | 3 +- .../src/appendixes/version-compatibility.md | 1 + .../1163-parallel-split-below-10-21.mdl | 64 +++++++++++ .../bug-tests/1163-workflow-on-mendix9.mdl | 6 +- modelsdk/canon/versionshape.go | 49 ++++++++- .../mpr/writer_version_shape_1163_test.go | 102 ++++++++++++++++++ sdk/workflows/workflow.go | 5 + 8 files changed, 225 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-parallel-split-marker-below-10-21.json create mode 100644 mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-parallel-split-marker-below-10-21.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-parallel-split-marker-below-10-21.json new file mode 100644 index 000000000..56b20bd0e --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-parallel-split-marker-below-10-21.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.x and 10.0–10.20 a workflow with a `parallel split` made the project unloadable: exec reported \"Created workflow\", then mx check (9.24.30, 10.12.19) died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$EndOfParallelSplitPathActivity`", "cause": "The builder (workflows.EndParallelSplitPath) and the ALTER mutator (InsertPath) end every path with Workflows$EndOfParallelSplitPathActivity, which 10.21+ needs to run the path. The TYPE is introduced in 10.21.0 (mendixmodelsdk: EndOfParallelSplitPathActivity.versionInfo). The generated version data holds property floors only, so neither the floor nor any gate ever compared a written $Type with the project's version", "file": "`modelsdk/canon/versionshape.go` (LowerToProjectShape: dropEndOfPathMarkers on every Workflows$Flow below 10.21, endOfParallelSplitPathSince)", "insight": "**For a type newer than the project, find the older shape in a Marketplace SAMPLE, not a module**: `mxcli marketplace search workflow` lists Type=Sample apps whose .mpk holds a whole Studio Pro-authored .mpr; \"Example project workflow extension\" 2.0.0 (117680, Studio Pro 9.4.0) has a parallel split whose ParallelSplitOutcome is `{$ID,$Type,Flow}` and whose paths hold their activities and nothing else — exactly what the lowering produces. Workflow Commons, Employee Onboarding (9.1.1) and Multilevel Dynamic Workflow (9.24.3) hold workflows but no split. **The convert-up oracle does not reconstruct writer-only markers**: `mx convert -p` 10.24.24 of the lowered 9.24 project added 16 PersistentIds, BoundaryEvents and OnWorkflowEvent but no end-of-path marker, and 10.24's mx check is clean on it — so it cannot tell you whether 10.21+ needs the marker (the 11.14 runtime measurement in workflows.EndParallelSplitPath does), and a Studio Pro 9 split upgraded that way is unmarked too. Lower strictly below 10.21. **Lower, don't gate**: the older shape is exact and measured, unlike the published OData service, which is a different document. **Audit what else the writers emit against the SDK's type-level `introduced`** (a text extractor over gen/*.js — never execute it): the same run found user tasks written as Workflows$SingleUserTaskActivity (10.12; Studio Pro 9 stores Workflows$UserTask — 9.24 TypeCacheUnknownTypeException) and user targeting under UserTargeting (11.2, declared on the abstract UserTaskActivity so the floor misses it): on 10.12.19 and 10.24.24 a `targeting xpath […]` user task is CE1859 \"XPath constraint is required\", and mx convert 10.12 rewrites it to UserSource with the constraint EMPTIED. Not fixed here. Measured: bug-test on fresh 9.24.30 and 10.12.19 apps → before TypeCacheUnknownTypeException, after 0 errors, mx convert changes no key; describe → exec on 9.24 \"Unchanged workflow\"; 10.24.24 and 11.13.0 keep 4 markers, 0 errors, key sets identical before/after. Control: stubbing the lowering fails TestNormalize_ParallelSplitPathsBelowMendix10_21 with the marker on 9.24 and 10.20. mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl", "refs": ["#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 9734de71e..007339543 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A page passing `$currentObject` to a microflow or nanoflow is written on Mendix 9 and on 10.0–10.9** (ako/mxcli#1163) — `call microflow M.F(P = $currentObject)` on a button or a data grid column was refused with "cannot store Forms$MicroflowParameterMapping.Expression (Mendix 10.10.0+)". An argument held as an expression arrived in 10.10. Before it, Studio Pro stores no argument for the enclosing object and infers it. Without the refusal, 9.24 could not even load the page, because the mapping had no `Variable`. Below 10.10 that argument is now left out, and a page parameter argument is stored as before. Any other expression argument is refused, naming the parameter. - **A constant override in a configuration is written on Mendix 9 and on 10.0–10.8** (ako/mxcli#1163) — `alter settings constant @M.C value '…' in configuration 'Default'` was refused with "cannot store Settings$ConstantValue.SharedOrPrivateValue (Mendix 10.9.0+)". Before 10.9 an override's value is a flat `Value` string; Studio Pro 9.24 does not know the nested form and empties the override on load. A new override is now written flat there, and an existing one is updated in its own shape. A private value has no form before 10.9 and is refused, naming the constant. - **A workflow written to Mendix 9.24 has Studio Pro's property set** (ako/mxcli#1163) — its two state-change event handlers (`WorkflowOnStateChangeEvent`, `UsertaskOnStateChangeEvent`, removed in 11.0) were completed only from 10.24, where they were measured. `mx convert` 9.24.30 adds both as well, and Mendix's merge engine refuses to compare documents whose property names differ (#1373). -- **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. A `parallel split` is still unwritable below 10.21: its paths end in an activity type that version does not have. +- **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. +- **A workflow with a `parallel split` no longer makes a Mendix 9 or 10.0–10.20 project unloadable** (ako/mxcli#1163) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$EndOfParallelSplitPathActivity". mxcli ends every path with that marker, because from 10.21 the workflow engine runs a path only up to it, but the type itself arrived in 10.21. Before 10.21 Studio Pro stores a path as its activities and nothing more (measured on a Studio Pro 9.4 app). Below 10.21 the marker is no longer written, for `create`, `create or modify` and a path `alter workflow` inserts. Measured on 9.24.30 and 10.12.19: 0 errors. On 9.24, `describe` → `exec` reports the workflow unchanged. 10.21 and later are unchanged. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index e6b4cd22a..78e924507 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -186,6 +186,7 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | A text box's, text area's or date picker's placeholder | 10.11 | `Placeholder` text, instead of a `PlaceholderTemplate` | | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | +| The end of a parallel split path (`Workflows$EndOfParallelSplitPathActivity`, a type introduced in 10.21) | 10.21 | Not stored: a path holds its activities and nothing more (Studio Pro 9.4) | | A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | | A page's flow call argument that is the enclosing object (`call microflow M.F(P = $currentObject)`, `call nanoflow …`) | 10.10 | Not stored: Studio Pro infers it. A page parameter, snippet parameter or widget argument is stored as before, as a `Variable` | diff --git a/mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl b/mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl new file mode 100644 index 000000000..fc0f86d91 --- /dev/null +++ b/mdl-examples/bug-tests/1163-parallel-split-below-10-21.mdl @@ -0,0 +1,64 @@ +mdl 1; +-- ako/mxcli#1163: on Mendix 9.x and on 10.0–10.20 a workflow with a +-- `parallel split` made the project unloadable. `exec` reported success, then +-- mx check (and Studio Pro) could not load the project at all: +-- +-- TypeCacheUnknownTypeException: The type cache does not contain a type +-- with qualified name Workflows$EndOfParallelSplitPathActivity. +-- +-- mxcli ends every parallel split path with that marker, because from 10.21 +-- the workflow engine runs a path only up to it. The type itself arrived in +-- 10.21 (mendixmodelsdk), and the property floor checks property versions, +-- never a type's own. Before 10.21 Studio Pro stores a path as its activities +-- and nothing else (Studio Pro 9.4.0: the Marketplace's "Example project +-- workflow extension" 2.0.0, a split whose two paths hold a user task each), +-- so below 10.21 the marker is no longer written. +-- +-- Measured on fresh apps (`mxcli new … --skip-build`), exec then `mxcli docker +-- check`: +-- 9.24.30.51827 before: TypeCacheUnknownTypeException; after: 0 errors. +-- `mx convert -p` 9.24.30 changes no key; describe → exec +-- prints "Unchanged workflow". +-- 10.12.19.81612 before: TypeCacheUnknownTypeException; after: 0 errors; +-- `mx convert -p` 10.12.19 changes no key. +-- 10.24.24, 11.13.0 every path still ends in the marker, 0 errors, the +-- same key sets as before the fix. +-- +-- The ALTER below inserts a path through the mutator (InsertPath), outside +-- CREATE's encoder; the lowering runs on every unit write, so it covers both. + +create module Probe1163Ps; + +create persistent entity Probe1163Ps.Request ( + Title: String(100), + Amount: Integer +); + +create microflow Probe1163Ps.ACT_Left ($WorkflowContext: Probe1163Ps.Request) +begin + log info node 'Probe1163Ps' 'left'; +end; + +create microflow Probe1163Ps.ACT_Right ($WorkflowContext: Probe1163Ps.Request) +begin + log info node 'Probe1163Ps' 'right'; +end; + +create workflow Probe1163Ps.WF_Split + parameter $WorkflowContext: Probe1163Ps.Request + display 'Split' +begin + call microflow Probe1163Ps.ACT_Left; + parallel split + path 1 { call microflow Probe1163Ps.ACT_Left; } + path 2 { call microflow Probe1163Ps.ACT_Right; } + path 3 { } + ; + call microflow Probe1163Ps.ACT_Right; +end workflow; + +alter workflow Probe1163Ps.WF_Split { + insert into 'Parallel split' { path { + call microflow Probe1163Ps.ACT_Right; + } } +}; diff --git a/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl b/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl index ecea673ad..6d2529ef7 100644 --- a/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl +++ b/mdl-examples/bug-tests/1163-workflow-on-mendix9.mdl @@ -16,9 +16,9 @@ mdl 1; -- then `mxcli docker check` reports the blank app's error count, and `mx -- convert -p` strips no PersistentId. -- --- Not covered: a `parallel split`. Its paths end in a --- Workflows$EndOfParallelSplitPathActivity, a type Mendix introduced in 10.21, --- so below 10.21 the project does not load (TypeCacheUnknownTypeException). +-- A `parallel split` is in 1163-parallel-split-below-10-21.mdl: its paths end +-- in a Workflows$EndOfParallelSplitPathActivity, a type Mendix introduced in +-- 10.21, which below 10.21 is no longer written. create module Probe1163Wf; diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 0ee201248..7ac90cb7b 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -53,6 +53,12 @@ import ( // WorkflowActivity, Outcome and BoundaryEvent (ako/mxcli#1163). Workflows // exist from 9.0; `mx convert` 9.24.30 strips every one. The workflow's // own was refused; the others slipped past the floor. +// - Workflows$EndOfParallelSplitPathActivity, a TYPE introduced in 10.21.0, +// which ends every parallel split path mxcli writes (ako/mxcli#1163). +// Before, a path is its activities and nothing more (Studio Pro 9.4.0, the +// Marketplace's "Example project workflow extension" 2.0.0), and a project +// holding the marker does not load (TypeCacheUnknownTypeException). The +// floor checks property versions, never a type's own. // // It runs on every unit write, before the floor, so no writer needs to know // these shapes. What the older shape cannot hold — a design property value @@ -72,9 +78,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { noPersistentID: !PropertyDeclared("Workflows$Workflow", "PersistentId", *pv), flatConstant: !PropertyDeclared("Settings$ConstantValue", "SharedOrPrivateValue", *pv), variableArgs: !PropertyDeclared("Forms$MicroflowParameterMapping", "Expression", *pv), + unmarkedPaths: pv.Compare(endOfParallelSplitPathSince) < 0, } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID && !l.flatConstant && !l.variableArgs { + !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths { return contents, nil } var doc bson.D @@ -95,10 +102,17 @@ type shapeLowering struct { pv version.Version flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool - flatConstant, variableArgs bool + flatConstant, variableArgs, unmarkedPaths bool changed bool } +// endOfParallelSplitPathSince is the version that introduced the type +// Workflows$EndOfParallelSplitPathActivity (mendixmodelsdk 4.117.0: +// EndOfParallelSplitPathActivity.versionInfo introduced "10.21.0"). It is +// written down here because the generated version data holds property +// versions only, never a type's own. +var endOfParallelSplitPathSince = version.Version{Major: 10, Minor: 21} + var placeholderWidgets = map[string]bool{"Forms$TextBox": true, "Forms$TextArea": true, "Forms$DatePicker": true} // walk returns v with every element below it in the project's shape. @@ -140,6 +154,9 @@ func (l *shapeLowering) walk(v any) (any, error) { if l.noPersistentID && strings.HasPrefix(typ, "Workflows$") { x = l.dropPersistentID(x) } + if l.unmarkedPaths && typ == "Workflows$Flow" { + l.dropEndOfPathMarkers(x) + } if l.variableArgs { if err = l.lowerFlowArguments(x); err != nil { return nil, err @@ -378,6 +395,34 @@ func (l *shapeLowering) dropPersistentID(d bson.D) bson.D { return append(append(bson.D{}, d[:i]...), d[i+1:]...) } +// dropEndOfPathMarkers removes the Workflows$EndOfParallelSplitPathActivity +// from a flow's activities below 10.21, where the type does not exist. The +// list keeps its leading type marker, so a path that held only the end-of-path +// activity stores `Activities: [3]`, as Studio Pro 9 stores an empty flow. +// Nothing is refused: +// the marker is the writer's, never something a script states, and a path +// before 10.21 is its activities alone (Studio Pro 9.4.0's parallel split). +func (l *shapeLowering) dropEndOfPathMarkers(flow bson.D) { + i := shapeIndex(flow, "Activities") + if i < 0 { + return + } + list, _ := flow[i].Value.(bson.A) + out := make(bson.A, 0, len(list)) + for _, el := range list { + if d, ok := el.(bson.D); ok { + if t, _ := shapeString(d, "$Type"); t == "Workflows$EndOfParallelSplitPathActivity" { + continue + } + } + out = append(out, el) + } + if len(out) != len(list) { + flow[i].Value = out + l.changed = true + } +} + // listItems returns the elements of a stored list, without its type marker. func listItems(v any) bson.A { a, _ := v.(bson.A) diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index 3d33cfcb7..1542401b5 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -164,6 +164,108 @@ func TestNormalize_WorkflowPersistentIDsBelowMendix10_21(t *testing.T) { } } +// endOfPath is the marker mxcli ends a parallel split path with +// (workflows.EndParallelSplitPath). +func endOfPath() bson.D { + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$EndOfParallelSplitPathActivity"}, + {Key: "Annotation", Value: nil}, {Key: "Caption", Value: "End of parallel split path"}, + {Key: "Name", Value: "EndOfParallelSplitPath"}, {Key: "PersistentId", Value: pid()}} +} + +// workflowWithParallelSplit is a workflow as mxcli builds it: a parallel split +// whose first path calls a microflow and whose second is empty, each ending in +// the end-of-path marker, followed by another microflow call. +func workflowWithParallelSplit() bson.D { + call := func(name string) bson.D { + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$CallMicroflowTask"}, + {Key: "BoundaryEvents", Value: bson.A{int32(2)}}, {Key: "Caption", Value: name}, + {Key: "Microflow", Value: "M." + name}, {Key: "Name", Value: name}, {Key: "PersistentId", Value: pid()}} + } + path := func(acts ...bson.D) bson.D { + list := bson.A{int32(3)} + for _, a := range acts { + list = append(list, a) + } + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$ParallelSplitOutcome"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: list}}}, + {Key: "PersistentId", Value: pid()}} + } + split := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$ParallelSplitActivity"}, + {Key: "Annotation", Value: nil}, {Key: "Caption", Value: "Parallel split"}, {Key: "Name", Value: "split"}, + {Key: "Outcomes", Value: bson.A{int32(3), path(call("Left"), endOfPath()), path(endOfPath())}}, + {Key: "PersistentId", Value: pid()}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), split, call("After")}}}}, + {Key: "Name", Value: "WF"}, + {Key: "PersistentId", Value: pid()}, + {Key: "Title", Value: "WF"}} +} + +// pathActivityTypes returns, per parallel split path, the $Type of each +// activity its flow holds. +func pathActivityTypes(t *testing.T, d bson.D) [][]string { + t.Helper() + outcomes, _ := get(find(d, "Workflows$ParallelSplitActivity"), "Outcomes") + var out [][]string + for _, o := range listItems(outcomes) { + flow, _ := get(o.(bson.D), "Flow") + acts, _ := get(flow.(bson.D), "Activities") + if a, _ := acts.(bson.A); len(a) == 0 || a[0] != int32(3) { + t.Errorf("a path's Activities is %v, want a list with marker 3", acts) + } + types := []string{} + for _, a := range listItems(acts) { + typ, _ := get(a.(bson.D), "$Type") + types = append(types, typ.(string)) + } + out = append(out, types) + } + return out +} + +// Workflows$EndOfParallelSplitPathActivity is introduced in 10.21.0 +// (mendixmodelsdk: the type's own versionInfo). mxcli ends every parallel split +// path with one, because from 10.21 the engine runs a path only up to it — and +// below 10.21 the type does not exist: mx check 9.24.30 does not load the +// project at all ("TypeCacheUnknownTypeException: The type cache does not +// contain a type with qualified name Workflows$EndOfParallelSplitPathActivity"). +// The property floor holds property versions only, so it never saw the type. +// Studio Pro before 10.21 stores a path as its activities and nothing else: +// the Marketplace's "Example project workflow extension" 2.0.0 (Studio Pro +// 9.4.0) has a split whose two paths hold one user task each, no marker, and +// an empty flow stores `Activities: [3]`. So below 10.21 the marker is not +// written; from 10.21 it is, and must be. +func TestNormalize_ParallelSplitPathsBelowMendix10_21(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + want string + }{ + {pv(9, 24, 30), "Workflows$CallMicroflowTask|"}, + {pv(10, 20, 0), "Workflows$CallMicroflowTask|"}, + {pv(10, 21, 0), "Workflows$CallMicroflowTask,Workflows$EndOfParallelSplitPathActivity|Workflows$EndOfParallelSplitPathActivity"}, + {pv(11, 13, 0), "Workflows$CallMicroflowTask,Workflows$EndOfParallelSplitPathActivity|Workflows$EndOfParallelSplitPathActivity"}, // control + } { + d, err := normalizeShape(t, tt.v, workflowWithParallelSplit()) + if err != nil { + t.Errorf("%d.%d: a parallel split is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + var paths []string + for _, p := range pathActivityTypes(t, d) { + paths = append(paths, strings.Join(p, ",")) + } + if got := strings.Join(paths, "|"); got != tt.want { + t.Errorf("%d.%d: paths hold %s, want %s", tt.v.MajorVersion, tt.v.MinorVersion, got, tt.want) + } + // The rest of the workflow is untouched: the call after the split stays. + acts, _ := get(find(d, "Workflows$Flow"), "Activities") + if n := len(listItems(acts)); n != 2 { + t.Errorf("%d.%d: the main flow holds %d activities, want 2", tt.v.MajorVersion, tt.v.MinorVersion, n) + } + } +} + // settingsWithConstantValue is the project settings unit as `alter settings // constant … in configuration` writes a new override: the value nested in a // Settings$SharedValue (or a Settings$PrivateValue, which holds none). flat is diff --git a/sdk/workflows/workflow.go b/sdk/workflows/workflow.go index dc57904ce..0f6fa8a82 100644 --- a/sdk/workflows/workflow.go +++ b/sdk/workflows/workflow.go @@ -395,6 +395,11 @@ func (a *EndOfParallelSplitPathActivity) ActivityType() string { return "EndOfPa // the path (a jump, or an end-of-workflow), where a marker would be unreachable. // A nil flow — an empty path — gets a flow holding only the marker, which is how // an empty path is stored. +// +// The marker's type is introduced in Mendix 10.21; before it a path is its +// activities alone, and a project holding the marker does not load. The marker +// is added on every version here and dropped below 10.21 on the way to storage +// (canon.LowerToProjectShape), so the builders need not know the version. func EndParallelSplitPath(flow *Flow, newID func() model.ID) *Flow { if flow == nil { flow = &Flow{} From 761fab054f5589e292352f7037bd64d485b53f42 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 28 Oct 2026 22:56:43 +0000 Subject: [PATCH 24/44] fix(modelsdk): give WidgetReturnType.assignableTo and GroupBox.headerMode their floors Symptom (ako/mxcli#1163): on Mendix 9.0-9.19 every pluggable widget's schema lost the Type of its return types and kept an AssignableTo that version does not have (49 return types across the embedded templates). A return type reaching the floor there was refused with "this Mendix 9.18.0 project cannot store CustomWidgets$WidgetReturnType.Type (Mendix 9.20.0+)". Cause: the vendored generator gave a property the row of the next property in the SDK's versionInfo and left that one out, as it did AggregateFunction. mendixmodelsdk 4.117.0 has WidgetReturnType `type: {}`, `assignableTo: {introduced 9.20.0}`, and pages.GroupBox `caption: {}`, `headerMode: {introduced 8.10.0}`. widgetTypeForVersion drops undeclared keys from the widget schema by the same data, so it dropped the right key. Evidence: Studio Pro stores {EntityProperty, IsList, Type} on 208 of 208 return types in a 9.1.1 app and 326 of 326 in a 9.4.0 app (Marketplace samples), and adds AssignableTo on 57 of 57 in a 9.24.3 one. The group box row changes nothing mxcli writes: the caption is stored as CaptionTemplate, which the table never keyed, and 8.10 is below every supported version. Workflows$CallMicroflowTask.boundaryEvents was already 10.14.0, as in the SDK, and stays. Measured on 10.12.19 with that floor removed: mxcli writes BoundaryEvents [2] on every call and mx convert strips it, so the floor's drop-empty is right. A test now pins it; removing the row fails it. Fix: correct both rows by hand and regenerate modelsdk/version. Only those two rows change in the copy. Measured: the bug-test on fresh 10.24.24 and 11.13.0 apps checks at 0 errors before and after, with identical key sets. 9.24.30 stores Studio Pro 9.24.3's return type keys both times (its mx check stops on a separate pre-existing NullReferenceException in WidgetValue.IsDefault). Control: the new tests fail on the old rows (49 wrong key sets at 9.18; the Type refusal). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...type-and-group-box-rows-misattributed.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1163-widget-return-type-below-9-20.mdl | 61 +++++++++ .../modelsdk/widget_template_version_test.go | 69 ++++++++++ modelsdk/gen/customwidgets/version.go | 8 +- modelsdk/gen/pages/version.go | 12 +- .../mpr/writer_version_shape_1163_test.go | 128 ++++++++++++++++++ modelsdk/version/metamodel_versions_gen.go | 20 ++- 9 files changed, 294 insertions(+), 8 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-28-widget-return-type-and-group-box-rows-misattributed.json create mode 100644 mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-widget-return-type-and-group-box-rows-misattributed.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-widget-return-type-and-group-box-rows-misattributed.json new file mode 100644 index 000000000..46deef833 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-28-widget-return-type-and-group-box-rows-misattributed.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-28", "symptom": "On Mendix 9.0–9.19 every pluggable widget's schema lost the Type of its return types and kept an AssignableTo the version does not have (49 return types across the embedded templates); a CustomWidgets$WidgetReturnType reaching the floor there was refused: `this Mendix 9.18.0 project cannot store CustomWidgets$WidgetReturnType.Type (Mendix 9.20.0+)`. Forms$GroupBox had caption at 8.10 and no row for headerMode", "cause": "The vendored generator (modelsdk/gen/*/version.go) gave a property the row of the NEXT property in the SDK's versionInfo and left that one out, as it did Microflows$AggregateAction.aggregateFunction: mendixmodelsdk 4.117.0 has WidgetReturnType `type: {}`, `assignableTo: {introduced 9.20.0}` and pages.GroupBox `caption: {}`, `headerMode: {introduced 8.10.0}`. canon.PropertyDeclared reads the same table, and widgetTypeForVersion uses it to drop undeclared keys from the embedded 11.6 widget schema, so the wrong row dropped the right key", "file": "`modelsdk/gen/customwidgets/version.go`, `modelsdk/gen/pages/version.go` (fixed by hand), `modelsdk/version/metamodel_versions_gen.go` (go generate ./modelsdk/version)", "insight": "**Compare the whole table with the SDK, then read each hit's neighbour**: a text extractor over the SDK's gen/*.js StructureVersionInfo (map Pages$ to Forms$ for storage names; never execute the SDK) found 50 differences; most are SDK data newer than the vendored generator (11.11–11.16 rows it lacks), and five are this misattribution, always 'property gets its neighbour's version, neighbour missing': AggregateAction.aggregateFunction, WidgetReturnType.type, GroupBox.caption, and on Deleted LayoutGridRow.columns (← class, 8.0 — the row propertyfloor.go cites as why generated deletions are not trusted) and DynamicImageViewer.showAsThumbnail (← onClickBehavior, 7.18), plus a never-looked-up SDK-named duplicate row Microflows$AggregateListAction. The Deleted ones and the duplicate change nothing the floor does and were left. Workflows$CallMicroflowTask.boundaryEvents was NOT a mismatch (10.14.0 on both sides): the earlier list's fourth row was a false positive. **Check which BSON key a row is looked up by before predicting its effect**: the floor keys by the lower-cased BSON key, and a group box's caption is stored as CaptionTemplate, so the wrong caption row never applied to anything; headerMode's 8.10 is below every version mxcli writes — the GroupBox fix corrects the data and changes no write. **Studio Pro-authored bytes settle a row faster than a platform run when no mxbuild of that version is cached**: Marketplace samples carry whole apps — Employee Onboarding (9.1.1) stores {EntityProperty,IsList,Type} on 208 of 208 return types, Example project workflow extension 2.0.0 (9.4.0) on 326 of 326, Multilevel Dynamic Workflow (9.24.3) adds AssignableTo on 57 of 57. For boundaryEvents the oracle was a floor-off build on 10.12.19: mxcli then writes `BoundaryEvents: [2]` on every call, mx check passes, mx convert 10.12.19 strips it — the existing floor's drop-empty is what keeps the write clean, and a stated boundary event is refused. Tests: TestTemplateWidgetReturnTypeKeysAcrossMendix9_20 (fails on the old row with 49 wrong key sets at 9.18), TestNormalize_WidgetReturnTypeAcrossMendix9_20, TestPropertyDeclared_RowsTheGeneratorMisattributed, TestNormalize_CallMicroflowBoundaryEventsBelowMendix10_14 (control: removing the row fails it). Seen in passing, not fixed: on 9.24.30 a page with a slider or a data grid 2, written by mxcli, stops mx check (and its update-widgets step) with `NullReferenceException at WidgetValue.IsDefault() (WidgetValue.cs:879)`, with or without these fixes; 10.24.24 and 11.13.0 check the same pages at 0 errors. Bug-test: mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl", "refs": ["#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 007339543..ea06bf20b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Workflows are written on Mendix 9 and on 10.0–10.20** (ako/mxcli#1163) — every workflow was refused with "cannot store Workflows$Workflow.PersistentId (Mendix 10.21.0+)". Workflows exist from 9.0, but the persistent ID on a workflow, its activities, outcomes and boundary events arrived in 10.21. The activities' and outcomes' IDs were never refused, and `mx convert` 9.24 stripped them. mxcli mints these IDs itself, so below 10.21 it no longer writes them, for `create`, `create or modify` and the activities `alter workflow` inserts. - **A workflow with a `parallel split` no longer makes a Mendix 9 or 10.0–10.20 project unloadable** (ako/mxcli#1163) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$EndOfParallelSplitPathActivity". mxcli ends every path with that marker, because from 10.21 the workflow engine runs a path only up to it, but the type itself arrived in 10.21. Before 10.21 Studio Pro stores a path as its activities and nothing more (measured on a Studio Pro 9.4 app). Below 10.21 the marker is no longer written, for `create`, `create or modify` and a path `alter workflow` inserts. Measured on 9.24.30 and 10.12.19: 0 errors. On 9.24, `describe` → `exec` reports the workflow unchanged. 10.21 and later are unchanged. - **Aggregates in microflows are written on Mendix 9** (ako/mxcli#1163) — every `count`, `sum`, `average`, `minimum` and `maximum` on a 9.x project was refused with "cannot store Microflows$AggregateAction.AggregateFunction (Mendix 10.0.0+)". The version data had the aggregate's function introduced in 10.0, which belongs to the fold `reduce` added. Studio Pro 9.24 stores it on every aggregate. An aggregate over an expression, `reduce`, `all` and `any` are Mendix 10.0 and are still refused below it, now naming the fold. +- **Pluggable widgets keep the `Type` of their return types on Mendix 9.0–9.19** (ako/mxcli#1163) — the version data mxcli vendors from the Model SDK had a return type's `Type` introduced in 9.20. That version belongs to `AssignableTo`, which the data left out. A widget schema written to a 9.0–9.19 project therefore lost the `Type` of every return type and kept an `AssignableTo` that version does not have. A return type that reached the property floor was refused with "cannot store CustomWidgets$WidgetReturnType.Type (Mendix 9.20.0+)". Studio Pro 9.1 and 9.4 store `Type` on every return type and no `AssignableTo`. The same mistake gave a group box's caption the 8.10 version of its header mode. That row changed nothing mxcli writes, because the caption is stored under another name and mxcli supports no version that old. - **Pages on Mendix 9.24 and 10.x are read and written in the version's own shape** (ako/mxcli#1209) — several page properties are stored differently below a version: a design property's value before 10.2, a data view's page parameter before 10.0, its editability before 10.7, and a text box's placeholder before 10.11. `describe` read only the newer form, so on 9.24 a describe → exec round trip removed every design property and placeholder. Writing any page with a data view or a design property was refused, and `alter styling` on a stored design property reported success and changed nothing. Every write now converts to the project's form, and `describe` reads both. What the older form cannot hold is refused, naming the widget: a Spacing or colour design property, conditional editability, a placeholder with parameters. - **A text box's `KeyboardType` survives describe → exec** (ako/mxcli#1209) — the keyboard a native app opens had no MDL spelling and was written as `Default`, so a rewrite reset it. `textbox t (Attribute: Email, KeyboardType: EmailAddress)` (`Default`, `NumberPad`, `DecimalPad`, `EmailAddress`, `PhonePad`, `URL`); an unknown value is MDL-WIDGET18. - **A button's workflow action survives a rewrite** (ako/mxcli#832) — `describe` printed no action for Show user task page, Show workflow admin page or Call workflow, so re-executing a page turned them into no action (on ako/TestApp's TaskInbox and TaskDashboard, 2 of 6 user-task buttons each). They are now `open user task [with (AssignOnOpen, OpenWhenAssigned)]`, `open workflow with (DefaultPage: M.Page)` and `call workflow M.WF [with (…)]`. `complete task` takes `with (ClosePage, Commit)` instead of always writing both true. `open workflow` without a page is CE1834 and is refused. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 78e924507..8350ab77b 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -207,6 +207,8 @@ A published OData service is not an older shape of the same document. Before 10. Pluggable widget PropertyTypes change between Mendix versions. A widget template extracted from 11.6 may have more or fewer properties than the widget installed in a 10.24 project. The [widget augmentation system](../internals/widget-templates.md) handles this by reconciling templates against the project's `.mpk` files. +The schema itself also gained keys over time. A key the project's version does not declare is left out of the schema mxcli writes, by the same version data the property floor uses. For example, a return type's `AssignableTo` arrived in 9.20, so a 9.0–9.19 schema stores `EntityProperty`, `IsList` and `Type`, as Studio Pro 9.1 and 9.4 do ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). + ## Widget Template Versions Embedded widget templates are extracted from Mendix 11.6 and cover: diff --git a/mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl b/mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl new file mode 100644 index 000000000..42451968f --- /dev/null +++ b/mdl-examples/bug-tests/1163-widget-return-type-below-9-20.mdl @@ -0,0 +1,61 @@ +mdl 1; +-- ako/mxcli#1163: two more rows of the version data mxcli vendors from the +-- Model SDK gave a property the version of the property after it, and left +-- that one out — the same generator mistake as AggregateFunction: +-- +-- CustomWidgets$WidgetReturnType Type 9.20.0 (belongs to AssignableTo) +-- Forms$GroupBox Caption 8.10.0 (belongs to HeaderMode) +-- +-- mendixmodelsdk 4.117.0: WidgetReturnType `type: {}`, `assignableTo: +-- {introduced: "9.20.0"}`; pages.GroupBox `caption: {}`, `headerMode: +-- {introduced: "8.10.0"}`. +-- +-- What it changed: a pluggable widget's schema is written with only the keys +-- the project's version declares, by the same data. On 9.0–9.19 every return +-- type in it lost its Type and kept AssignableTo (49 return types across the +-- embedded templates; the slider below has 3). Studio Pro stores +-- {EntityProperty, IsList, Type} on 9.1.1 (208 of 208, the Marketplace's +-- Employee Onboarding app) and 9.4.0 (326 of 326, "Example project workflow +-- extension" 2.0.0), and {AssignableTo, EntityProperty, IsList, Type} on +-- 9.24.3 (57 of 57, Multilevel Dynamic Workflow). The group box row changed +-- nothing mxcli writes: the caption is stored as CaptionTemplate, which the +-- table never keyed, and 8.10 is below every version mxcli supports. +-- +-- Measured, exec then `mxcli docker check` on fresh apps, before and after: +-- 10.24.24 and 11.13.0 report 0 errors and store the same key sets. 9.24.30 +-- stores Studio Pro 9.24.3's return type key set both times, but its mx check +-- stops with a NullReferenceException in WidgetValue.IsDefault on this page, +-- and on a data grid 2 page. That happens with the build before either +-- ako/mxcli#1163 fix as well, so it is a separate defect. Not run on a +-- 9.0–9.19 platform: those mxbuilds are Mono builds (mx.exe), and no Mono +-- runtime is installed. + +create module Probe1163Rt; + +create enumeration Probe1163Rt.Country ( + NL 'Netherlands', + BE 'Belgium' +); + +create persistent entity Probe1163Rt.Customer ( + Name: String(100), + Country: enumeration(Probe1163Rt.Country), + Score: Integer +); + +create page Probe1163Rt.Customer_Edit +( + params: ( $Customer: Probe1163Rt.Customer ), + title: 'Customer', + layout: Atlas_Core.PopupLayout +) +{ + dataview dvCustomer (datasource: $Customer) { + groupbox gbCustomer (caption: 'Customer', HeaderMode: H3, Collapsible: YesExpanded) { + textbox txtName (label: 'Name', attribute: Name) + pluggablewidget 'com.mendix.widget.custom.slider.Slider' sldScore ( + valueAttribute: Score + ) + } + } +}; diff --git a/mdl/backend/modelsdk/widget_template_version_test.go b/mdl/backend/modelsdk/widget_template_version_test.go index ea7789649..7edeea39d 100644 --- a/mdl/backend/modelsdk/widget_template_version_test.go +++ b/mdl/backend/modelsdk/widget_template_version_test.go @@ -184,6 +184,75 @@ func TestTemplateWidgetValueTypeKeysAt10_15(t *testing.T) { } } +// typeKeySets returns the key set of every element of typeName in an encoded +// document. +func typeKeySets(t *testing.T, b []byte, typeName string) [][]string { + t.Helper() + var doc bsonv2.D + if err := bsonv2.Unmarshal(b, &doc); err != nil { + t.Fatal(err) + } + var out [][]string + var walk func(v any) + walk = func(v any) { + switch x := v.(type) { + case bsonv2.D: + var keys []string + match := false + for _, e := range x { + keys = append(keys, e.Key) + if e.Key == "$Type" && e.Value == typeName { + match = true + } + walk(e.Value) + } + if match { + sort.Strings(keys) + out = append(out, keys) + } + case bsonv2.A: + for _, e := range x { + walk(e) + } + } + } + walk(doc) + return out +} + +// CustomWidgets$WidgetReturnType: Type on every version, AssignableTo from +// 9.20.0 (mendixmodelsdk 4.117.0). The vendored generator gave Type the row of +// AssignableTo and left AssignableTo out, so on 9.0–9.19 the template's return +// types lost the Type every one of them holds and kept an AssignableTo the +// version does not have (ako/mxcli#1163). Studio Pro stores {EntityProperty, +// IsList, Type} on 9.1.1 (208 of 208, the Marketplace's Employee Onboarding +// app) and 9.4.0 (326 of 326, "Example project workflow extension" 2.0.0), and +// adds AssignableTo by 9.24.3 (57 of 57, Multilevel Dynamic Workflow 1.0.0). +func TestTemplateWidgetReturnTypeKeysAcrossMendix9_20(t *testing.T) { + for _, tt := range []struct { + pv *types.ProjectVersion + want string + }{ + {v(9, 18, 0), "$ID,$Type,EntityProperty,IsList,Type"}, + {v(9, 20, 0), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, + {v(9, 24, 30), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, + {v(11, 13, 0), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, // control + } { + seen := 0 + for _, id := range mwidgets.ListAvailableTemplates() { + for _, got := range typeKeySets(t, encodeTemplateWidget(t, id, tt.pv), "CustomWidgets$WidgetReturnType") { + seen++ + if strings.Join(got, ",") != tt.want { + t.Errorf("%d.%d %s: WidgetReturnType keys %v, want %s", tt.pv.MajorVersion, tt.pv.MinorVersion, id, got, tt.want) + } + } + } + if seen == 0 { + t.Fatalf("%d.%d: no template holds a WidgetReturnType", tt.pv.MajorVersion, tt.pv.MinorVersion) + } + } +} + func containsKey(s []string, k string) bool { for _, x := range s { if x == k { diff --git a/modelsdk/gen/customwidgets/version.go b/modelsdk/gen/customwidgets/version.go index de213ddd9..101778f48 100644 --- a/modelsdk/gen/customwidgets/version.go +++ b/modelsdk/gen/customwidgets/version.go @@ -48,7 +48,13 @@ var VersionInfos = map[string]version.TypeVersionInfo{ }, "CustomWidgets$WidgetReturnType": { Properties: map[string]version.PropertyVersionInfo{ - "type": {Introduced: "9.20.0"}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave type + // the row of assignableTo, which it then left out. mendixmodelsdk + // 4.117.0 has type with no version and assignableTo introduced + // 9.20.0; Studio Pro 9.1.1 and 9.4.0 store Type and no AssignableTo + // on every return type, Studio Pro 9.24.3 both. + "assignableTo": {Introduced: "9.20.0"}, + "type": {}, }, }, "CustomWidgets$WidgetValue": { diff --git a/modelsdk/gen/pages/version.go b/modelsdk/gen/pages/version.go index 506d0cc1d..eab4e5d9c 100644 --- a/modelsdk/gen/pages/version.go +++ b/modelsdk/gen/pages/version.go @@ -406,9 +406,15 @@ var VersionInfos = map[string]version.TypeVersionInfo{ }, "Forms$GroupBox": { Properties: map[string]version.PropertyVersionInfo{ - "caption": {Introduced: "8.10.0"}, - "widget": {Deleted: "7.15.0"}, - "widgets": {Introduced: "7.15.0"}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave caption + // the row of headerMode, which it then left out. mendixmodelsdk + // 4.117.0 has caption with no version and headerMode introduced + // 8.10.0; Studio Pro 9.1.1, 9.4.0 and 9.24.3 store both (the caption + // as CaptionTemplate). + "caption": {}, + "headerMode": {Introduced: "8.10.0"}, + "widget": {Deleted: "7.15.0"}, + "widgets": {Introduced: "7.15.0"}, }, }, "Forms$Header": { diff --git a/modelsdk/mpr/writer_version_shape_1163_test.go b/modelsdk/mpr/writer_version_shape_1163_test.go index 1542401b5..db8be67f5 100644 --- a/modelsdk/mpr/writer_version_shape_1163_test.go +++ b/modelsdk/mpr/writer_version_shape_1163_test.go @@ -8,7 +8,9 @@ import ( "go.mongodb.org/mongo-driver/v2/bson" + "github.com/mendixlabs/mxcli/modelsdk/canon" "github.com/mendixlabs/mxcli/modelsdk/mpr/version" + mversion "github.com/mendixlabs/mxcli/modelsdk/version" ) // ako/mxcli#1163: what the 9.24 doctype run found mxcli writing that Mendix 9 @@ -82,6 +84,132 @@ func TestNormalize_ReduceFoldIsRefusedBelowMendix10(t *testing.T) { } } +// The vendored generator gave two more properties the row of their type's next +// property and left that one out, as it did AggregateFunction: mendixmodelsdk +// 4.117.0 has CustomWidgets$WidgetReturnType `type: {}` and `assignableTo: +// {introduced: "9.20.0"}`, and Forms$GroupBox (pages.GroupBox) `caption: {}` +// and `headerMode: {introduced: "8.10.0"}`. The table had type 9.20.0 and +// caption 8.10.0, and no row for assignableTo or headerMode. +// Workflows$CallMicroflowTask.boundaryEvents is 10.14.0 in both, and stays. +func TestPropertyDeclared_RowsTheGeneratorMisattributed(t *testing.T) { + for _, tt := range []struct { + typ, key string + v *version.ProjectVersion + want bool + }{ + // Studio Pro 9.1.1 and 9.4.0 store Type on every return type; 9.24.3 + // adds AssignableTo. + {"CustomWidgets$WidgetReturnType", "Type", pv(9, 0, 0), true}, + {"CustomWidgets$WidgetReturnType", "Type", pv(9, 18, 0), true}, + {"CustomWidgets$WidgetReturnType", "AssignableTo", pv(9, 19, 0), false}, + {"CustomWidgets$WidgetReturnType", "AssignableTo", pv(9, 20, 0), true}, + // The caption is stored as CaptionTemplate, which the table never + // keyed: the wrong row applied to nothing. HeaderMode is 8.10, below + // every version mxcli writes. + {"Forms$GroupBox", "CaptionTemplate", pv(8, 0, 0), true}, + {"Forms$GroupBox", "HeaderMode", pv(8, 9, 0), false}, + {"Forms$GroupBox", "HeaderMode", pv(8, 10, 0), true}, + {"Forms$GroupBox", "HeaderMode", pv(9, 0, 0), true}, + // Unchanged: right in the table already. + {"Workflows$CallMicroflowTask", "BoundaryEvents", pv(10, 13, 0), false}, + {"Workflows$CallMicroflowTask", "BoundaryEvents", pv(10, 14, 0), true}, + } { + v := mversion.Version{Major: tt.v.MajorVersion, Minor: tt.v.MinorVersion, Patch: tt.v.PatchVersion} + if got := canon.PropertyDeclared(tt.typ, tt.key, v); got != tt.want { + t.Errorf("%s.%s declared on %s = %v, want %v", tt.typ, tt.key, v, got, tt.want) + } + } +} + +// pageWithReturnType is a page holding one pluggable widget whose Type has a +// return type as the template carries it: Type set, AssignableTo empty. +func pageWithReturnType() bson.D { + rt := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$WidgetReturnType"}, + {Key: "AssignableTo", Value: ""}, {Key: "EntityProperty", Value: ""}, {Key: "IsList", Value: false}, {Key: "Type", Value: "String"}} + vt := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$WidgetValueType"}, {Key: "ReturnType", Value: rt}} + widget := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$CustomWidget"}, {Key: "Name", Value: "w"}, + {Key: "Type", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$CustomWidgetType"}, + {Key: "ObjectType", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$WidgetObjectType"}, + {Key: "PropertyTypes", Value: bson.A{int32(2), bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "CustomWidgets$WidgetPropertyType"}, + {Key: "ValueType", Value: vt}}}}}}}}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Forms$Page"}, {Key: "Name", Value: "P"}, + {Key: "Widgets", Value: bson.A{int32(2), widget}}} +} + +// Below 9.20 a return type's Type was refused ("this Mendix 9.18.0 project +// cannot store CustomWidgets$WidgetReturnType.Type (Mendix 9.20.0+)") and its +// AssignableTo passed. Now Type is kept and the empty AssignableTo dropped. +func TestNormalize_WidgetReturnTypeAcrossMendix9_20(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + want string + }{ + {pv(9, 18, 0), "$ID,$Type,EntityProperty,IsList,Type"}, + {pv(9, 20, 0), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, + {pv(11, 13, 0), "$ID,$Type,AssignableTo,EntityProperty,IsList,Type"}, // control + } { + d, err := normalizeShape(t, tt.v, pageWithReturnType()) + if err != nil { + t.Errorf("%d.%d: a widget with a return type is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + if got := strings.Join(keysOf(find(d, "CustomWidgets$WidgetReturnType")), ","); got != tt.want { + t.Errorf("%d.%d: return type keys %s, want %s", tt.v.MajorVersion, tt.v.MinorVersion, got, tt.want) + } + } +} + +// boundaryEvent is a timer boundary event as mxcli writes one (10.20's type). +func boundaryEvent() bson.D { + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$InterruptingTimerBoundaryEvent"}, + {Key: "Caption", Value: "Late"}, {Key: "FirstExecutionTime", Value: "addDays([%CurrentDateTime%], 1)"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: bson.A{int32(3)}}}}} +} + +// workflowWithCallBoundaryEvents is a workflow holding one microflow call with +// the given BoundaryEvents list. +func workflowWithCallBoundaryEvents(events bson.A) bson.D { + call := bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$CallMicroflowTask"}, + {Key: "BoundaryEvents", Value: events}, {Key: "Caption", Value: "Validate"}, + {Key: "Microflow", Value: "M.Validate"}, {Key: "Name", Value: "call"}} + return bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: bid()}, {Key: "$Type", Value: "Workflows$Flow"}, + {Key: "Activities", Value: bson.A{int32(3), call}}}}, + {Key: "Name", Value: "WF"}, {Key: "Title", Value: "WF"}} +} + +// Workflows$CallMicroflowTask.BoundaryEvents is 10.14.0 (mendixmodelsdk, and +// the table). The writer gives every microflow call an empty list, so below +// 10.14 the empty list is dropped, not refused: Studio Pro 9.24.3 stores no +// BoundaryEvents on a call (Multilevel Dynamic Workflow 1.0.0), and with the +// row's floor removed mxcli writes `BoundaryEvents: [2]` on 10.12.19, which +// `mx convert` 10.12.19 strips. A boundary event the script states is +// refused there, naming the key. +func TestNormalize_CallMicroflowBoundaryEventsBelowMendix10_14(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + kept bool + }{ + {pv(9, 24, 30), false}, + {pv(10, 13, 0), false}, + {pv(10, 14, 0), true}, + {pv(11, 13, 0), true}, // control + } { + d, err := normalizeShape(t, tt.v, workflowWithCallBoundaryEvents(bson.A{int32(2)})) + if err != nil { + t.Errorf("%d.%d: a microflow call is refused: %v", tt.v.MajorVersion, tt.v.MinorVersion, err) + continue + } + if _, ok := get(find(d, "Workflows$CallMicroflowTask"), "BoundaryEvents"); ok != tt.kept { + t.Errorf("%d.%d: BoundaryEvents kept %v, want %v", tt.v.MajorVersion, tt.v.MinorVersion, ok, tt.kept) + } + } + _, err := normalizeShape(t, pv(10, 13, 0), workflowWithCallBoundaryEvents(bson.A{int32(2), boundaryEvent()})) + if err == nil || !strings.Contains(err.Error(), "Workflows$CallMicroflowTask.BoundaryEvents (Mendix 10.14.0+)") { + t.Errorf("a boundary event on a microflow call on 10.13 was not refused naming the key: %v", err) + } +} + // pid is a fresh PersistentId: a non-zero GUID, as the writer mints one (a zero // GUID is an empty value, which the floor would drop rather than refuse). func pid() bson.Binary { diff --git a/modelsdk/version/metamodel_versions_gen.go b/modelsdk/version/metamodel_versions_gen.go index 37591339c..899146f87 100644 --- a/modelsdk/version/metamodel_versions_gen.go +++ b/modelsdk/version/metamodel_versions_gen.go @@ -341,7 +341,13 @@ var metamodelVersions = []map[string]TypeVersionInfo{ }, "CustomWidgets$WidgetReturnType": { Properties: map[string]PropertyVersionInfo{ - "type": {Introduced: "9.20.0"}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave type + // the row of assignableTo, which it then left out. mendixmodelsdk + // 4.117.0 has type with no version and assignableTo introduced + // 9.20.0; Studio Pro 9.1.1 and 9.4.0 store Type and no AssignableTo + // on every return type, Studio Pro 9.24.3 both. + "assignableTo": {Introduced: "9.20.0"}, + "type": {}, }, }, "CustomWidgets$WidgetValue": { @@ -2416,9 +2422,15 @@ var metamodelVersions = []map[string]TypeVersionInfo{ }, "Forms$GroupBox": { Properties: map[string]PropertyVersionInfo{ - "caption": {Introduced: "8.10.0"}, - "widget": {Deleted: "7.15.0"}, - "widgets": {Introduced: "7.15.0"}, + // Fixed by hand (ako/mxcli#1163): the vendored generator gave caption + // the row of headerMode, which it then left out. mendixmodelsdk + // 4.117.0 has caption with no version and headerMode introduced + // 8.10.0; Studio Pro 9.1.1, 9.4.0 and 9.24.3 store both (the caption + // as CaptionTemplate). + "caption": {}, + "headerMode": {Introduced: "8.10.0"}, + "widget": {Deleted: "7.15.0"}, + "widgets": {Introduced: "7.15.0"}, }, }, "Forms$Header": { From 4939d516d8d8fe8d52893cf3a8eff5ba42bfd886 Mon Sep 17 00:00:00 2001 From: engalar Date: Thu, 29 Oct 2026 09:00:59 +0800 Subject: [PATCH 25/44] =?UTF-8?q?fix(git):=20doctor=20judges=20only=20core?= =?UTF-8?q?.autocrlf=20=E2=80=94=20the=20mendix.*=20counters=20are=20Studi?= =?UTF-8?q?o=20Pro's=20own=20(measured=20on=20the=20lab=20repo,=20S0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- cmd/mxcli/cmd_git_doctor.go | 5 +- cmd/mxcli/gitdoctor.go | 59 +++++++++++++++-------- cmd/mxcli/gitdoctor_test.go | 41 +++++++++++++++- docs-site/src/tools/outside-studio-pro.md | 2 +- 4 files changed, 82 insertions(+), 25 deletions(-) diff --git a/cmd/mxcli/cmd_git_doctor.go b/cmd/mxcli/cmd_git_doctor.go index 1e17d23f3..10d5d61b4 100644 --- a/cmd/mxcli/cmd_git_doctor.go +++ b/cmd/mxcli/cmd_git_doctor.go @@ -23,8 +23,9 @@ var gitDoctorCmd = &cobra.Command{ Checked: 1. Local git config: core.autocrlf=False (a checkout must not rewrite line - endings inside .mxunit files), mendix.commits-since-gc and - mendix.lineEndingResetDone (Studio Pro's own bookkeeping). --fix sets them. + endings inside .mxunit files; --fix rewrites any other value), and the + mendix.* keys (Studio Pro's own bookkeeping) backfilled when missing — + a value Studio Pro wrote stays. 2. The remote URL. Studio Pro can only use HTTPS; an SSH remote (git@host:org/app.git or ssh://…) is rewritten to HTTPS by --fix. Other non-HTTPS URLs are reported without being changed. diff --git a/cmd/mxcli/gitdoctor.go b/cmd/mxcli/gitdoctor.go index 48f045728..c0726d4b0 100644 --- a/cmd/mxcli/gitdoctor.go +++ b/cmd/mxcli/gitdoctor.go @@ -31,11 +31,23 @@ import ( var errDoctorIssuesRemain = errors.New("issues remain (re-run with --fix to repair)") -// mxConfigKeys is the local git config Studio Pro writes and expects. -var mxConfigKeys = [][2]string{ - {"core.autocrlf", "False"}, - {"mendix.commits-since-gc", "0"}, - {"mendix.lineEndingResetDone", "True"}, +// mxConfigKey is one entry of the local git config Studio Pro writes. +// fallback is what a missing key is backfilled with. presentValuePasses marks +// the keys whose value is Studio Pro's own bookkeeping — measured on a real +// checkout (the mx-git-lab lab, S0), commits-since-gc counts up between gc +// runs and lineEndingResetDone comes back in any case, so only a MISSING key +// is an issue there. core.autocrlf is judged: true (or input) makes a +// checkout rewrite line endings inside .mxunit files on Windows. +type mxConfigKey struct { + key string + fallback string + presentValuePasses bool +} + +var mxConfigKeys = []mxConfigKey{ + {"core.autocrlf", "False", false}, + {"mendix.commits-since-gc", "0", true}, + {"mendix.lineEndingResetDone", "True", true}, } var ( @@ -52,25 +64,32 @@ func doctorRun(root, mprPath, remoteOverride, version string, fix bool, out io.W found, fixed := 0, 0 fmt.Fprintln(out, "Git local config") - for _, kv := range mxConfigKeys { - key, want := kv[0], kv[1] + for _, ck := range mxConfigKeys { + key, want := ck.key, ck.fallback current := gitConfigLocal(root, key) - if strings.EqualFold(current, want) { - fmt.Fprintf(out, " [x] %s = %s\n", key, current) - continue - } - found++ - if current == "" { + switch { + case current == "": + found++ fmt.Fprintf(out, " [ ] %s — missing (expected %s)\n", key, want) - } else { + if fix { + if _, err := gitOut(root, "config", "--local", key, want); err != nil { + return fmt.Errorf("set %s: %w", key, err) + } + fixed++ + fmt.Fprintf(out, " set to %s\n", want) + } + case ck.presentValuePasses || strings.EqualFold(current, "false"): + fmt.Fprintf(out, " [x] %s = %s\n", key, current) + default: + found++ fmt.Fprintf(out, " [ ] %s = %s (expected %s)\n", key, current, want) - } - if fix { - if _, err := gitOut(root, "config", "--local", key, want); err != nil { - return fmt.Errorf("set %s: %w", key, err) + if fix { + if _, err := gitOut(root, "config", "--local", key, want); err != nil { + return fmt.Errorf("set %s: %w", key, err) + } + fixed++ + fmt.Fprintf(out, " set to %s\n", want) } - fixed++ - fmt.Fprintf(out, " set to %s\n", want) } } diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index c0e0393f4..fafe0a9b0 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -58,8 +58,7 @@ func TestGitDoctor_ConfigMissingReported(t *testing.T) { } } -func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { - dir := t.TempDir() +func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { dir := t.TempDir() doctorEnv(t, dir) gitIn(t, dir, "init", "-q", "-b", "main") @@ -87,6 +86,44 @@ func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { } } +// Measured on a real Studio Pro checkout (the mx-git-lab lab, S0): Studio Pro +// writes core.autocrlf=false in lowercase, counts commits-since-gc up between +// garbage collections (2 after two commits) and writes lineendingresetdone in +// lowercase. A healthy Studio Pro checkout must pass untouched. +func TestGitDoctor_StudioProValuesPass(t *testing.T) { + dir := t.TempDir() + doctorEnv(t, dir) + gitIn(t, dir, "init", "-q", "-b", "main") + gitIn(t, dir, "config", "--local", "core.autocrlf", "false") + gitIn(t, dir, "config", "--local", "mendix.commits-since-gc", "2") + gitIn(t, dir, "config", "--local", "mendix.lineendingresetdone", "true") + + var out bytes.Buffer + if err := doctorRun(dir, "", "", "", false, &out); err != nil { + t.Fatalf("a healthy Studio Pro checkout must pass: %v\n%s", err, out.String()) + } + if strings.Contains(out.String(), "mendix.commits-since-gc — ") { + t.Errorf("the gc counter must not be judged:\n%s", out.String()) + } +} + +// core.autocrlf=true is the one config value that damages a Mendix checkout: +// a checkout rewrites line endings inside .mxunit files on Windows. +func TestGitDoctor_AutoCRLFTrueIsFixed(t *testing.T) { + dir := t.TempDir() + doctorEnv(t, dir) + gitIn(t, dir, "init", "-q", "-b", "main") + gitIn(t, dir, "config", "--local", "core.autocrlf", "true") + + var out bytes.Buffer + if err := doctorRun(dir, "", "", "", true, &out); err != nil { + t.Fatalf("--fix must repair autocrlf=true: %v\n%s", err, out.String()) + } + if got := configLocal(t, dir, "core.autocrlf"); !strings.EqualFold(got, "false") { + t.Errorf("core.autocrlf = %q, want false", got) + } +} + // --- doctor: remote URL scheme ---------------------------------------------- func TestGitDoctor_SSHRemoteRewrittenByFix(t *testing.T) { diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 9cf80d53a..6b71c0b19 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -224,7 +224,7 @@ mxcli git doctor --fix --version 10.21.0 | Check | `--fix` | |-------|---------| -| local git config: `core.autocrlf=False` (a checkout must not rewrite line endings inside `.mxunit` files), `mendix.commits-since-gc`, `mendix.lineEndingResetDone` | sets them | +| local git config: `core.autocrlf=False` (a checkout must not rewrite line endings inside `.mxunit` files; any other value is rewritten), `mendix.commits-since-gc`, `mendix.lineEndingResetDone` — the `mendix.*` values are Studio Pro's own bookkeeping and are only backfilled when missing | sets/backs them | | the remote URL: Studio Pro can only use HTTPS | rewrites SSH (`git@host:org/app.git`, `ssh://…`) to HTTPS; other non-HTTPS URLs are reported unchanged | | the remote's `refs/notes/mx_metadata` | reported only — push the notes ref together with the branch | | each commit on the checked-out branch and its mx_metadata note: missing notes make Studio Pro show `(unknown)`; a malformed note it cannot read at all | computes real notes from the model (`-p`), falling back to a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); when no version is known the newest valid note in history supplies it | From 6ddd5e5ed709dea826e3b84ddaf6e32fee96af99 Mon Sep 17 00:00:00 2001 From: engalar Date: Thu, 29 Oct 2026 09:11:50 +0800 Subject: [PATCH 26/44] fix(git): the staged-files summary lists what the commit holds, not the junk just unstaged (lab finding F4) --- cmd/mxcli/cmd_git_commit.go | 20 +++++++++++++------- cmd/mxcli/gitdoctor_test.go | 27 ++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 8 deletions(-) diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index cbdb52cbe..bf9fadcb7 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -138,16 +138,22 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error fmt.Fprintln(out, "nothing to commit, working tree clean") return nil } - fmt.Fprintf(out, "Staged %d file(s):\n", len(staged)-len(junk)) - kept := staged - if len(kept) > 10 { - kept = kept[:10] + kept := make([]string, 0, len(staged)-len(junk)) + for _, f := range staged { + if !isJunkPath(f) { + kept = append(kept, f) + } } - for _, f := range kept { + fmt.Fprintf(out, "Staged %d file(s):\n", len(kept)) + shown := kept + if len(shown) > 10 { + shown = shown[:10] + } + for _, f := range shown { fmt.Fprintf(out, " %s\n", f) } - if len(staged)-len(junk) > 10 { - fmt.Fprintf(out, " … and %d more\n", len(staged)-len(junk)-10) + if len(kept) > 10 { + fmt.Fprintf(out, " … and %d more\n", len(kept)-10) } if dryRun { diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index fafe0a9b0..2d43138c9 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -577,9 +577,34 @@ func TestGitCommit_DryRunLeavesHashDriftUntouched(t *testing.T) { } } -func TestGitCommit_DryRunWithPushRefused(t *testing.T) { +// The staged-files summary must list what the commit will hold, not the junk +// that was just unstaged (measured on the mx-git-lab lab: the list opened with +// 10 .mxcli/ paths while the commit held 2 model files). +func TestGitCommit_StagedListOmitsJunk(t *testing.T) { dir, mprPath := doctorNoteRepo(t) doctorEnv(t, dir) + units := baseUnits(t) + units[tPage] = tUnit{tFolder, unitDoc(t, "Forms$Page", "Home", bson.E{Key: "Title", Value: "List"})} + writeProject(t, dir, "11.13.0", units) + if err := os.WriteFile(filepath.Join(dir, "project-settings.user.json"), []byte("{}"), 0o644); err != nil { + t.Fatal(err) + } + + var out bytes.Buffer + if err := commitRun(mprPath, "clean list", false, false, &out); err != nil { + t.Fatalf("commitRun: %v\n%s", err, out.String()) + } + section := out.String()[strings.Index(out.String(), "Staged"):] + if strings.Contains(section, "project-settings.user.json") { + t.Errorf("the staged list names junk that was left out:\n%s", section) + } + if !strings.Contains(section, "App.mpr") { + t.Errorf("the staged list must name the model files:\n%s", section) + } +} + +func TestGitCommit_DryRunWithPushRefused(t *testing.T) { dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) var out bytes.Buffer if err := commitRun(mprPath, "both", true, true, &out); err == nil { From 4e961b23b2a81c3be7d902ba892cbeafa18d61b6 Mon Sep 17 00:00:00 2001 From: engalar Date: Thu, 29 Oct 2026 10:43:10 +0800 Subject: [PATCH 27/44] =?UTF-8?q?feat(git):=20`git=20merge`=20reviews=20co?= =?UTF-8?q?nflicts=20before=20they=20land=20=E2=80=94=20report=20both=20si?= =?UTF-8?q?des,=20then=20--ours/--theirs=20or=20Studio=20Pro=20(lab=20S5b)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .claude/skills/mendix/teamserver-git/SKILL.md | 23 +- CHANGELOG.md | 1 + cmd/mxcli/cmd_git_commit.go | 11 +- cmd/mxcli/cmd_git_merge.go | 416 ++++++++++++++++++ cmd/mxcli/gitdoctor_test.go | 6 +- cmd/mxcli/gitmerge_test.go | 305 +++++++++++++ docs-site/src/tools/outside-studio-pro.md | 38 ++ 7 files changed, 780 insertions(+), 20 deletions(-) create mode 100644 cmd/mxcli/cmd_git_merge.go create mode 100644 cmd/mxcli/gitmerge_test.go diff --git a/.claude/skills/mendix/teamserver-git/SKILL.md b/.claude/skills/mendix/teamserver-git/SKILL.md index 3b2af2afd..ea51485aa 100644 --- a/.claude/skills/mendix/teamserver-git/SKILL.md +++ b/.claude/skills/mendix/teamserver-git/SKILL.md @@ -14,9 +14,9 @@ description: "Commit, branch and push a Mendix project in git the way Studio Pro | 42-55 | Branches: create them on the server | | 56-97 | Commit from the agent | | 98-112 | Push the branch and the notes together, and soon | -| 113-125 | Never rewrite or merge model history with git | -| 126-136 | Reading what Studio Pro recorded | -| 137-145 | Checklist | +| 113-132 | Never rewrite or merge model history with git | +| 133-143 | Reading what Studio Pro recorded | +| 144-152 | Checklist | # Working with Git on a Mendix Team Server Project @@ -112,11 +112,18 @@ git push origin refs/notes/mx_metadata ## Never rewrite or merge model history with git -- **No `git merge`, `rebase` or `cherry-pick` of model changes.** The `.mpr` is - a SQLite file; git cannot merge it, and a textual merge of `mprcontents/` - leaves an index that does not match the files. Studio Pro has a model-aware - merge: **Version Control › Merge Changes Here** (*Merge feature branch* or - *Cherry Pick*). Studio Pro's history on Team Server is linear for that reason. +- **No plain `git merge`, `rebase` or `cherry-pick` of model changes.** The + `.mpr` is a SQLite file; git cannot merge it (every two-sided model change + is a binary conflict), and a textual merge of `mprcontents/` leaves an + index that does not match the files. Model-aware merging exists in two + forms: **`mxcli git merge `** from the command line (it hands the + conflict to `mx git-merge`, Studio Pro's own engine, and commits the + resolved model as a normal merge commit with the note), and **Version + Control › Merge Changes Here** in Studio Pro (*Merge feature branch* or + *Cherry Pick*) — which is also where the conflicts mx cannot auto-resolve + go. A conflict `mxcli git merge` cannot resolve aborts the merge cleanly. + Studio Pro's history on Team Server is linear for a reason: do not "fix" + a conflicted model merge by hand-editing files. - **No `--amend` or force-push of a pushed commit.** Before pushing, an amend is fine: the `notes.rewriteRef` Studio Pro puts in `.git/config` carries the note. Re-run `mxcli git note --write` anyway; it is cheap. diff --git a/CHANGELOG.md b/CHANGELOG.md index ea7ab00ce..5a9751d20 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. +- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT THEIRS_COMMIT`. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three projects to `mx git-merge`, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything mx auto-resolves is staged and committed as the normal two-parent merge commit with the mx_metadata note — measured end-to-end on a two-sided domain-model change, `AutoResolved`, the merge commit held both sides and the note matched Studio Pro's own merge in shape; a conflict mx reports as `ManualResolutionRequired` is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit (ours and theirs, overlapping units marked); the human then re-runs with `--ours` or `--theirs` to take one side of every conflicted model file (deterministic, mx not consulted again), or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; non-model conflicts (text sources) are left in place like plain git. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index bf9fadcb7..697c6ef84 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -179,16 +179,7 @@ func commitRun(mprPath, message string, push, dryRun bool, out io.Writer) error return fmt.Errorf("committed %s, but attaching the note failed (%v) — run 'mxcli git note -p %s --write' to attach it", shortCommit(head), err, mprPath) } - if push { - if branch == "" { - return errors.New("detached HEAD; cannot push — check out a branch and push manually") - } - if _, err := gitOutIn(root, nil, "push", "origin", branch, mxMetadataRef); err != nil { - return fmt.Errorf("push %s and %s: %w", branch, mxMetadataRef, err) - } - fmt.Fprintf(out, "Pushed %s and %s to origin.\n", branch, mxMetadataRef) - } - return nil + return pushBranchAndNotes(out, root, branch, push) } // nulFields splits git's -z output (NUL-separated, no quoting) into paths. diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go new file mode 100644 index 000000000..ecdcebc74 --- /dev/null +++ b/cmd/mxcli/cmd_git_merge.go @@ -0,0 +1,416 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strings" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/mendixlabs/mxcli/modelsdk/mpr" + "github.com/spf13/cobra" +) + +// cmd_git_merge.go: `mxcli git merge ` — merge a branch the way Studio +// Pro does. `git merge` runs first; when it stops on the model files (the +// .mpr is SQLite and every .mxunit is BSON, so any two-sided model change is +// a binary conflict), the conflict is handed to `mx git-merge` — the same +// model-aware engine Studio Pro's own merge uses — with the three revisions +// checked out into temporary worktrees. What mx resolves is committed as a +// normal two-parent merge commit with the mx_metadata note; what it reports +// as ManualResolutionRequired aborts the merge with the conflict list and a +// pointer to Studio Pro, leaving nothing behind. + +var gitMergeCmd = &cobra.Command{ + Use: "merge ", + Short: "Merge a branch with model-aware conflict resolution (mx git-merge)", + Long: `Merge into the current branch, resolving model conflicts the way +Studio Pro does. + +'git merge ' runs first. A merge git resolves itself (disjoint files, a +fast-forward) is committed and annotated like 'mxcli git commit' does. When +git stops on the model files — the .mpr and the .mxunit units are binary, so +two sides changing the model always conflict — the three revisions are +checked out into temporary worktrees and 'mx git-merge BASE MINE THEIRS' +(Studio Pro's own model-aware merge, resolved for the project's Mendix +version) decides: + + - everything mx auto-resolves is staged and committed as the merge commit, + with the mx_metadata note attached (this keeps both sides — the usual + best outcome); + - a conflict mx reports as ManualResolutionRequired is never merged + silently: the merge is aborted, and both sides' changes are printed for + review. Then decide: + --ours take our side of every conflicted model file + --theirs take their side + (deterministic; mx is not consulted again), or resolve for real in + Studio Pro (Version Control › Merge Changes Here), which stays the tool + for mixed resolutions; + - a conflict in non-model files (text sources) is left in place for manual + resolution, like plain git. + +The merge is refused while Studio Pro has the project open (.mpr.lock). +MPR v2 projects only. The mx binary is resolved for the project's Mendix +version (PATH, the mxcli cache, or the Studio Pro installation).`, + Example: ` mxcli git merge feature/orders -p app.mpr + mxcli git merge origin/main -p app.mpr --push + mxcli git merge feature/orders -p app.mpr --theirs # after reviewing the report`, + Args: cobra.ExactArgs(1), + RunE: runGitMerge, + SilenceUsage: true, + SilenceErrors: true, +} + +func init() { + gitMergeCmd.Flags().StringP("project", "p", "", "path to the Mendix project (.mpr)") + _ = gitMergeCmd.MarkFlagRequired("project") + gitMergeCmd.Flags().StringP("message", "m", "", "merge commit message (default: git's)") + gitMergeCmd.Flags().Bool("ours", false, "on a model conflict mx cannot resolve, take our side of every conflicted model file") + gitMergeCmd.Flags().Bool("theirs", false, "on a model conflict mx cannot resolve, take their side of every conflicted model file") + gitMergeCmd.Flags().Bool("push", false, "push the branch and the notes ref to their remote afterwards") + gitCmd.AddCommand(gitMergeCmd) +} + +func runGitMerge(cmd *cobra.Command, args []string) error { + ref := args[0] + message, _ := cmd.Flags().GetString("message") + ours, _ := cmd.Flags().GetBool("ours") + theirs, _ := cmd.Flags().GetBool("theirs") + push, _ := cmd.Flags().GetBool("push") + project, _ := cmd.Flags().GetString("project") + strategy := "" + switch { + case ours && theirs: + return errors.New("--ours and --theirs cannot be combined") + case ours: + strategy = "ours" + case theirs: + strategy = "theirs" + } + return mergeRun(project, ref, message, strategy, push, realMxGitMerge, cmd.OutOrStdout()) +} + +// mxMergeRunner runs `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT +// THEIRS_COMMIT` and returns mx's exit code and output. +type mxMergeRunner func(base, mine, theirs string, shas [3]string) (int, string, error) + +// mergeRun is the command without its wiring. runMx is injected for tests. +// mergeRun is the command without its wiring. runMx is injected for tests. +// +// Conflict handling is a three-step flow a human stays in charge of: +// +// 1. `git merge` runs; a merge it resolves itself is annotated and done. +// 2. A model conflict goes to `mx git-merge` first — what it auto-resolves +// keeps both sides, which is almost always the best outcome — and is +// committed with the note. +// 3. What mx reports as ManualResolutionRequired is never merged silently: +// with no strategy the merge is aborted and both sides' changes are +// printed for review; `--ours`/`--theirs` then take one side wholesale +// (mx is not consulted — the human has decided). +func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRunner, out io.Writer) error { + switch strategy { + case "", "ours", "theirs": + default: + return fmt.Errorf("unknown strategy %q (use --ours or --theirs)", strategy) + } + abs, err := filepath.Abs(mprPath) + if err != nil { + return err + } + if !mpr.AllowWritesWhileStudioProOpen && os.Getenv(mpr.AllowStudioProOpenEnv) != "1" { + if lock, mod := mpr.StudioProLockFile(abs); lock != "" { + return &mpr.StudioProOpenError{MprPath: abs, LockPath: lock, LockTime: mod} + } + } + repo, err := openGitNoteRepo(mprPath) + if err != nil { + return err + } + defer repo.Close() + root := repo.Root + + branch := strings.TrimSpace(gitOutOr(root, "symbolic-ref", "--quiet", "--short", "HEAD")) + if branch == "" { + return errors.New("detached HEAD: check out a branch before merging") + } + + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + if mergeErr == nil { + if strings.Contains(mergeOut, "Already up to date") { + fmt.Fprintln(out, "Already up to date.") + return nil + } + head := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + fmt.Fprintf(out, "Merged %s without model conflicts (%s).\n", ref, shortCommit(head)) + return finishMerge(out, repo, head, branch, push) + } + + // A ref that does not resolve fails before any conflict exists. + if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { + return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + } + + conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) + var model, other []string + for _, p := range conflicted { + if isModelPath(repo, p) { + model = append(model, p) + } else { + other = append(other, p) + } + } + if len(other) > 0 { + fmt.Fprintln(out, "Non-model conflicts, resolve by hand, then 'git add' and 'git commit':") + for _, p := range other { + fmt.Fprintf(out, " %s\n", p) + } + return errors.New("merge has non-model conflicts left for manual resolution") + } + + baseSha := strings.TrimSpace(gitOutOr(root, "merge-base", "HEAD", "MERGE_HEAD")) + oursSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + theirsSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "MERGE_HEAD")) + + // The human has decided: take one side of every conflicted model file. + // Deterministic, and mx is not consulted — `git checkout --ours/--theirs` + // reads the conflicted stages directly. + if strategy != "" { + side := "--" + strategy + for _, p := range model { + if _, err := gitOutIn(root, nil, "checkout", side, "--", p); err != nil { + return fmt.Errorf("take the %s version of %s: %w", strategy, p, err) + } + } + args := []string{"add", "--"} + args = append(args, model...) + if _, err := gitOutIn(root, nil, args...); err != nil { + return fmt.Errorf("stage the resolved model files: %w", err) + } + return commitTheMerge(root, repo, model, message, branch, push, out) + } + + // mx first: what it auto-resolves keeps both sides, which is almost + // always better than taking one side wholesale. + // Check out the base and theirs revisions; MINE is the working tree. + tmp, err := os.MkdirTemp("", "mxcli-gitmerge-") + if err != nil { + return err + } + baseDir := filepath.Join(tmp, "base") + theirsDir := filepath.Join(tmp, "theirs") + if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", baseDir, baseSha); err != nil { + os.RemoveAll(tmp) + return fmt.Errorf("check out the merge base: %w", err) + } + if _, err := gitOutIn(root, nil, "worktree", "add", "--detach", theirsDir, theirsSha); err != nil { + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) + os.RemoveAll(tmp) + return fmt.Errorf("check out %s: %w", ref, err) + } + cleanWorktrees := func() { + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", baseDir) + _, _ = gitOutIn(root, nil, "worktree", "remove", "--force", theirsDir) + _ = os.RemoveAll(tmp) + } + + version := projectMendixVersion(mprPath) + fmt.Fprintf(out, "Model files conflict (%s); merging the model with mx %s…\n", + strings.Join(shortNames(model), ", "), version) + code, mxOut, runErr := runMx( + filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), + abs, + filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), + [3]string{baseSha, oursSha, theirsSha}, + ) + if s := strings.TrimSpace(mxOut); s != "" { + fmt.Fprintln(out, s) + } + if runErr != nil { + cleanWorktrees() + return fmt.Errorf("mx git-merge failed: %w", runErr) + } + if code != 0 { + // The review step: show what each side changed, then leave nothing + // behind. The human decides — --ours/--theirs or Studio Pro. + cleanWorktrees() + printConflictReport(root, repo, baseSha, oursSha, theirsSha, out) + if _, abortErr := gitOutIn(root, nil, "merge", "--abort"); abortErr != nil { + return fmt.Errorf("the model merge has conflicts and aborting it failed (%v); resolve in Studio Pro", abortErr) + } + fmt.Fprintln(out, "Merge aborted; the working tree is back to the pre-merge state.") + return errors.New("the model merge needs a decision: re-run with --ours or --theirs to take one side wholesale, or resolve in Studio Pro (Version Control › Merge Changes Here)") + } + + args := []string{"add", "--"} + args = append(args, model...) + if _, err := gitOutIn(root, nil, args...); err != nil { + cleanWorktrees() + return fmt.Errorf("stage the resolved model files: %w", err) + } + if err := commitTheMerge(root, repo, nil, message, branch, push, out); err != nil { + cleanWorktrees() + return err + } + cleanWorktrees() + return nil +} + +// printConflictReport shows what each side changed — the review material a +// human needs before deciding how the conflict is resolved. +func printConflictReport(root string, repo *gitNoteRepo, baseSha, oursSha, theirsSha string, out io.Writer) { + base, err := repo.snapshot(baseSha) + if err != nil { + fmt.Fprintf(out, " (could not read the merge base: %v)\n", err) + return + } + ours, err := repo.snapshot(oursSha) + if err != nil { + fmt.Fprintf(out, " (could not read our side: %v)\n", err) + return + } + theirs, err := repo.snapshot(theirsSha) + if err != nil { + fmt.Fprintf(out, " (could not read their side: %v)\n", err) + return + } + oursChanges := computeModelChanges(base, ours, nil) + theirsChanges := computeModelChanges(base, theirs, nil) + theirsIDs := map[string]bool{} + for _, c := range theirsChanges { + theirsIDs[c.UnitID] = true + } + oursIDs := map[string]bool{} + for _, c := range oursChanges { + oursIDs[c.UnitID] = true + } + fmt.Fprintf(out, "\nBoth sides changed — a decision is needed on:\n") + for _, c := range oursChanges { + if theirsIDs[c.UnitID] { + fmt.Fprintf(out, " %s %s [%s] (unit %s)\n", c.UnitType, c.UnitName, c.Module, shortCommit(c.UnitID)) + } + } + fmt.Fprintf(out, "\nYours (%s):\n", shortCommit(oursSha)) + for _, c := range oursChanges { + fmt.Fprintf(out, " %-9s %s %s [%s]%s\n", c.Status, c.UnitType, c.UnitName, c.Module, oursTheirsMark(theirsIDs, c.UnitID)) + } + fmt.Fprintf(out, "\nTheirs (%s):\n", shortCommit(theirsSha)) + for _, c := range theirsChanges { + fmt.Fprintf(out, " %-9s %s %s [%s]%s\n", c.Status, c.UnitType, c.UnitName, c.Module, oursTheirsMark(oursIDs, c.UnitID)) + } +} + +func oursTheirsMark(other map[string]bool, unitID string) string { + if other[unitID] { + return " ◄ also changed on the other side" + } + return "" +} + +// commitTheMerge commits the resolved merge (MERGE_HEAD is present), writes +// the note and pushes. stagedPaths, when given, are staged first. +func commitTheMerge(root string, repo *gitNoteRepo, stagedPaths []string, message, branch string, push bool, out io.Writer) error { + if stagedPaths != nil { + args := []string{"add", "--"} + args = append(args, stagedPaths...) + if _, err := gitOutIn(root, nil, args...); err != nil { + return fmt.Errorf("stage the resolved model files: %w", err) + } + } + commitArgs := []string{"commit", "--no-edit"} + if message != "" { + commitArgs = []string{"commit", "-m", message} + } + if _, err := gitOutIn(root, nil, commitArgs...); err != nil { + return fmt.Errorf("commit the merge: %w", err) + } + + head := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + fmt.Fprintf(out, "Model merge committed as %s.\n", shortCommit(head)) + return finishMerge(out, repo, head, branch, push) +} + +// finishMerge attaches the mx_metadata note to the finished merge and prints +// (or, with push, runs) the branch-and-notes push. +func finishMerge(out io.Writer, repo *gitNoteRepo, head, branch string, push bool) error { + if err := annotateCommits(out, repo, []string{head}, branch, true, false); err != nil { + return fmt.Errorf("committed %s, but attaching the note failed (%v) — run 'mxcli git note -p … --write' to attach it", shortCommit(head), err) + } + return pushBranchAndNotes(out, repo.Root, branch, push) +} + +func pushBranchAndNotes(out io.Writer, root, branch string, push bool) error { + if push { + if branch == "" { + return errors.New("detached HEAD; cannot push — check out a branch and push manually") + } + if _, err := gitOutIn(root, nil, "push", "origin", branch, mxMetadataRef); err != nil { + return fmt.Errorf("push %s and %s: %w", branch, mxMetadataRef, err) + } + fmt.Fprintf(out, "Pushed %s and %s to origin.\n", branch, mxMetadataRef) + return nil + } + if branch != "" { + fmt.Fprintf(out, "Push with: git push origin %s %s\n", branch, mxMetadataRef) + } + return nil +} + +// isModelPath reports whether a repo-relative path is part of the model: +// the .mpr itself or an .mxunit under mprcontents. +func isModelPath(repo *gitNoteRepo, rel string) bool { + f := filepath.ToSlash(rel) + if f == repo.MPRPath { + return true + } + return strings.HasPrefix(f, repo.Contents+"/") && strings.HasSuffix(f, ".mxunit") +} + +// projectMendixVersion reads the project's _ProductVersion for the mx +// resolution; "" on any error (ResolveMxForVersion then falls back to PATH). +func projectMendixVersion(mprPath string) string { + r, err := mpr.Open(mprPath) + if err != nil { + return "" + } + defer r.Close() + v, err := r.GetMendixVersion() + if err != nil { + return "" + } + return v +} + +// realMxGitMerge shells out to `mx git-merge -v …`, resolved for the +// project's Mendix version. +func realMxGitMerge(base, mine, theirs string, shas [3]string) (int, string, error) { + version := projectMendixVersion(mine) + mxPath, err := docker.ResolveMxForVersion("", version) + if err != nil { + return 1, "", fmt.Errorf("no mx for Mendix %s: %w (model conflicts need it)", version, err) + } + cmd := exec.Command(mxPath, "git-merge", "-v", base, mine, theirs, shas[0], shas[1], shas[2]) + b, err := cmd.CombinedOutput() + code := 0 + if err != nil { + var ee *exec.ExitError + if errors.As(err, &ee) { + code = ee.ExitCode() + } else { + return 1, string(b), err + } + } + return code, string(b), nil +} + +func shortNames(paths []string) []string { + out := make([]string, 0, len(paths)) + for _, p := range paths { + out = append(out, filepath.Base(p)) + } + return out +} diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index 2d43138c9..a621e2569 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -58,7 +58,8 @@ func TestGitDoctor_ConfigMissingReported(t *testing.T) { } } -func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { dir := t.TempDir() +func TestGitDoctor_FixWritesConfigAndPasses(t *testing.T) { + dir := t.TempDir() doctorEnv(t, dir) gitIn(t, dir, "init", "-q", "-b", "main") @@ -603,7 +604,8 @@ func TestGitCommit_StagedListOmitsJunk(t *testing.T) { } } -func TestGitCommit_DryRunWithPushRefused(t *testing.T) { dir, mprPath := doctorNoteRepo(t) +func TestGitCommit_DryRunWithPushRefused(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) doctorEnv(t, dir) var out bytes.Buffer diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go new file mode 100644 index 000000000..a727bf953 --- /dev/null +++ b/cmd/mxcli/gitmerge_test.go @@ -0,0 +1,305 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +func bsonKeyValue(key, value string) bson.E { + return bson.E{Key: key, Value: value} +} + +// `mxcli git merge ` — merge a branch with model-aware resolution: git +// merge first, and on a model-file conflict `mx git-merge` (the same engine +// Studio Pro uses). The mx run is injected as runMx so the orchestration is +// testable without a Mendix installation; the real runner shells out to mx. + +// modelConflictRepo makes a repository whose merge of "side" into "feature" +// conflicts on both model files: each side changed the same DomainModel unit. +func modelConflictRepo(t *testing.T) (string, string) { + t.Helper() + dir, mprPath := doctorNoteRepo(t) + units := baseUnits(t) + units[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", mainSideKey)} + writeProject(t, dir, "11.13.0", units) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "main side") + gitIn(t, dir, "switch", "-q", "-c", "side", "HEAD~1") + side := baseUnits(t) + side[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", sideKey)} + writeProject(t, dir, "11.13.0", side) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "side change") + gitIn(t, dir, "switch", "-q", "feature") + _, err := gitInE(t, dir, "merge", "side") + if err != nil && !strings.Contains(err.Error(), "exit status") { + t.Fatalf("fixture: expected a conflicted merge, got %v", err) + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); !strings.Contains(uu, ".mxunit") { + t.Fatalf("fixture: expected unresolved model files, got %q", uu) + } + return dir, mprPath +} + +var ( + mainSideKey = bsonKeyValue("DocA", "main") + sideKey = bsonKeyValue("DocB", "side") +) + +func okRunner(out string) mxMergeRunner { + return func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 0, out, nil + } +} + +func TestGitMerge_CleanMergeTextFileWritesNote(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + gitIn(t, dir, "branch", "docs", "HEAD~1") + gitIn(t, dir, "switch", "-q", "docs") + if err := os.WriteFile(filepath.Join(dir, "NOTES.txt"), []byte("docs\n"), 0o644); err != nil { + t.Fatal(err) + } + gitIn(t, dir, "add", "NOTES.txt") + gitIn(t, dir, "commit", "-q", "-m", "docs side") + gitIn(t, dir, "switch", "-q", "feature") + + noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("mx must not run on a merge git resolves itself") + return 0, "", nil + } + var out bytes.Buffer + if err := mergeRun(mprPath, "docs", "", "", false, noMx, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + var n mxNote + if err := json.Unmarshal([]byte(noteOn(t, dir, "HEAD")), &n); err != nil { + t.Fatalf("merge commit has no valid note: %v", err) + } + if !n.HasModelerVersion || n.BranchName != "feature" { + t.Errorf("note header: %+v", n) + } +} + +func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + + var gotBase, gotMine, gotTheirs string + var gotShas [3]string + runner := func(base, mine, theirs string, shas [3]string) (int, string, error) { + gotBase, gotMine, gotTheirs, gotShas = base, mine, theirs, shas + return 0, "Merge successful.", nil + } + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", false, runner, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + if !strings.Contains(gotBase, "mx-git-lab") && !strings.Contains(filepath.Base(gotBase), ".mpr") { + // the base path must be an .mpr inside a temporary checkout + if filepath.Ext(gotBase) != ".mpr" { + t.Errorf("runner got base %q, want the base checkout's .mpr", gotBase) + } + } + if filepath.Base(gotMine) != "App.mpr" { + t.Errorf("runner's mine = %q, want the working .mpr", gotMine) + } + if filepath.Base(gotTheirs) != "App.mpr" { + t.Errorf("runner's theirs = %q, want the theirs checkout's .mpr", gotTheirs) + } + if gotShas[0] == "" || gotShas[1] == gotShas[0] || gotShas[2] == gotShas[1] { + t.Errorf("shas must be base/ours/theirs distinct: %v", gotShas) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U") + if strings.TrimSpace(uu) != "" { + t.Errorf("conflicts still unresolved:\n%s", uu) + } + if !strings.Contains(noteOn(t, dir, "HEAD"), `"HasModelerVersion":true`) { + t.Errorf("merge commit has no note:\n%s", noteOn(t, dir, "HEAD")) + } +} + +func TestGitMerge_ModelConflictUnresolvedAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + fails := func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil + } + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", false, fails, &out) + if err == nil || !strings.Contains(err.Error(), "Studio Pro") { + t.Fatalf("want a Studio Pro referral, got %v", err) + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved despite the abort: %q", got) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the merge was not aborted") + } + uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U") + if strings.TrimSpace(uu) != "" { + t.Errorf("conflicted files remain:\n%s", uu) + } +} + +func TestGitMerge_NonModelConflictLeftAlone(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + gitIn(t, dir, "checkout", "-q", "-b", "side", "HEAD~1") + if err := os.WriteFile(filepath.Join(dir, "NOTES.txt"), []byte("side\n"), 0o644); err != nil { + t.Fatal(err) + } + gitIn(t, dir, "add", "NOTES.txt") + gitIn(t, dir, "commit", "-q", "-m", "side notes") + gitIn(t, dir, "checkout", "-q", "feature") + if err := os.WriteFile(filepath.Join(dir, "NOTES.txt"), []byte("main\n"), 0o644); err != nil { + t.Fatal(err) + } + gitIn(t, dir, "add", "NOTES.txt") + gitIn(t, dir, "commit", "-q", "-m", "main notes") + _, err := gitInE(t, dir, "merge", "side") + if err != nil && !strings.Contains(err.Error(), "exit status") { + t.Fatalf("fixture: expected a conflict, got %v", err) + } + + calls := 0 + counting := func(base, mine, theirs string, shas [3]string) (int, string, error) { + calls++ + return 0, "", nil + } + var out bytes.Buffer + merr := mergeRun(mprPath, "side", "", "", false, counting, &out) + if merr == nil { + t.Fatal("a non-model conflict must be reported, not silently passed") + } + if calls != 0 { + t.Errorf("mx must not run for non-model conflicts (ran %d times)", calls) + } + uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U") + if !strings.Contains(uu, "NOTES.txt") { + t.Errorf("the conflicted text file must be left for manual resolution:\n%s", uu) + } + gitInE(t, dir, "merge", "--abort") // clean up the fixture +} + +func TestGitMerge_AlreadyUpToDate(t *testing.T) { + dir, mprPath := doctorNoteRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("mx must not run") + return 0, "", nil + } + var out bytes.Buffer + if err := mergeRun(mprPath, "HEAD~1", "", "", false, noMx, &out); err != nil { + t.Fatalf("mergeRun: %v", err) + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } + if !strings.Contains(out.String(), "up to date") { + t.Errorf("must say so:\n%s", out.String()) + } +} + +// The conflict report is the review material: on ManualResolutionRequired the +// merge is aborted and both sides' changes are printed, so a human can decide +// before anything lands. mx is not even needed for the report. +func TestGitMerge_ConflictReportShowsBothSidesAndAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil + } + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "--ours") { + t.Fatalf("the report must point at the resolution flags, got %v", err) + } + report := out.String() + for _, want := range []string{"DomainModels$DomainModel", "also changed on the other side", "Yours", "Theirs"} { + if !strings.Contains(report, want) { + t.Errorf("report must contain %q:\n%s", want, report) + } + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } +} + +// --ours / --theirs take the decided side for every conflicted model file, +// commit the merge and write the note — deterministically, without mx. +func TestGitMerge_SideTakingResolves(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + + noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("--ours must not consult mx") + return 0, "", nil + } + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "ours", false, noMx, &out); err != nil { + t.Fatalf("mergeRun --ours: %v\n%s", err, out.String()) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + if !strings.Contains(noteOn(t, dir, "HEAD"), `"HasModelerVersion":true`) { + t.Errorf("merge commit has no note") + } + page := filepath.Join(dir, "mprcontents", tDM[:2], tDM[2:4], tDM+".mxunit") + raw, err := os.ReadFile(page) + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(raw, []byte("DocA")) || bytes.Contains(raw, []byte("DocB")) { + t.Error("--ours must leave our side of the unit on disk") + } + + // --theirs flips to the other side, deterministically. + dir2, mprPath2 := modelConflictRepo(t) + theirs := func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("--theirs must not consult mx") + return 0, "", nil + } + var out2 bytes.Buffer + if err := mergeRun(mprPath2, "side", "", "theirs", false, theirs, &out2); err != nil { + t.Fatalf("mergeRun --theirs: %v\n%s", err, out2.String()) + } + page2 := filepath.Join(dir2, "mprcontents", tDM[:2], tDM[2:4], tDM+".mxunit") + raw2, err := os.ReadFile(page2) + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(raw2, []byte("DocB")) || bytes.Contains(raw2, []byte("DocA")) { + t.Error("--theirs must leave their side of the unit on disk") + } +} diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 6b71c0b19..bc6c67299 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -210,6 +210,44 @@ commit the conflict markers. `--dry-run` verifies and reports (including ContentsHash drift, which it does not repair — a dry run writes nothing), stages, reports what it would commit, and unstages again without committing. +## Merging branches: `mxcli git merge` + +`git merge` cannot merge Mendix models: the `.mpr` is SQLite and every +`.mxunit` is BSON, so two sides changing the model always meet a binary +`CONFLICT (content)` — there is nothing for git's textual merge to work on. +Studio Pro's *Merge Changes Here* resolves it with a model-aware merge, and +that engine ships as a command: `mx git-merge BASE MINE THEIRS …`. + +`mxcli git merge ` puts the two together: + +```bash +mxcli git merge feature/orders -p app.mpr +mxcli git merge origin/main -p app.mpr --push +``` + +`git merge ` runs first; a merge git resolves itself (disjoint files, +a fast-forward) is annotated with the mx_metadata note like `mxcli git +commit` does. On a model-file conflict the base and theirs revisions are +checked out into temporary worktrees and `mx git-merge` runs (resolved for +the project's Mendix version): + +- what mx auto-resolves is staged and committed as the normal two-parent + merge commit, with the note attached; +- a conflict mx reports as `ManualResolutionRequired` is never merged + silently: the merge is aborted, and **both sides' changes are printed for + review** — unit by unit, ours and theirs. Then decide: + `--ours` or `--theirs` takes one side of every conflicted model file + (deterministic; mx is not consulted again), or resolve for real in + Studio Pro (*Version Control › Merge Changes Here*), which stays the tool + for mixed resolutions; +- a conflict in non-model files (text sources) is left in place for manual + resolution, like plain git. + +Merging is still never done by plain `git merge` on a Mendix repository: use +`mxcli git merge` or Studio Pro, never a textual merge of `mprcontents/`. +Like every mxcli write, the command is refused while Studio Pro has the +project open. + ## Auditing a repository: `mxcli git doctor` A repository that has known only plain git misses several things Studio Pro From f75997910891b129ac0b245d055101efa20e2223 Mon Sep 17 00:00:00 2001 From: engalar Date: Thu, 29 Oct 2026 11:25:41 +0800 Subject: [PATCH 28/44] =?UTF-8?q?feat(git):=20per-unit=20conflict=20resolu?= =?UTF-8?q?tion=20=E2=80=94=20property-level=20report=20(mx=20diff)=20and?= =?UTF-8?q?=20a=20plan=20file=20the=20human=20edits=20before=20landing=20(?= =?UTF-8?q?lab=20S5c)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_merge.go | 367 ++++++++++++++++++---- cmd/mxcli/gitmerge_test.go | 192 +++++++++-- docs-site/src/tools/outside-studio-pro.md | 13 +- 4 files changed, 478 insertions(+), 96 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a9751d20..6b6be1f07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/`) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` stages, reports and unstages without committing — and without repairing anything, hash drift included, because a dry run writes nothing. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT THEIRS_COMMIT`. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three projects to `mx git-merge`, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything mx auto-resolves is staged and committed as the normal two-parent merge commit with the mx_metadata note — measured end-to-end on a two-sided domain-model change, `AutoResolved`, the merge commit held both sides and the note matched Studio Pro's own merge in shape; a conflict mx reports as `ManualResolutionRequired` is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit (ours and theirs, overlapping units marked); the human then re-runs with `--ours` or `--theirs` to take one side of every conflicted model file (deterministic, mx not consulted again), or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; non-model conflicts (text sources) are left in place like plain git. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. +- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT THEIRS_COMMIT`. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three projects to `mx git-merge`, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything mx auto-resolves is staged and committed as the normal two-parent merge commit with the mx_metadata note — measured end-to-end on a two-sided domain-model change, `AutoResolved`, the merge commit held both sides and the note matched Studio Pro's own merge in shape; a conflict mx reports as `ManualResolutionRequired` is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit (ours and theirs, overlapping units marked); the run also writes a plan template (`.merge-plan.json`) beside the project, and the human then re-runs with `--plan ` to resolve **per unit** — each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files (modified-on-both units only; added/deleted/moved goes to Studio Pro) — or with `--ours`/`--theirs` to take one side wholesale, or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; non-model conflicts (text sources) are left in place like plain git. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS, and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index ecdcebc74..8d4679e54 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -3,11 +3,14 @@ package main import ( + "bytes" + "encoding/json" "errors" "fmt" "io" "os" "os/exec" + "path" "path/filepath" "strings" @@ -45,12 +48,18 @@ version) decides: best outcome); - a conflict mx reports as ManualResolutionRequired is never merged silently: the merge is aborted, and both sides' changes are printed for - review. Then decide: - --ours take our side of every conflicted model file - --theirs take their side - (deterministic; mx is not consulted again), or resolve for real in - Studio Pro (Version Control › Merge Changes Here), which stays the tool - for mixed resolutions; + review — at property level, from 'mx diff' (e.g. + "New type > Length: 100 → 150" on ours, "→ 200" on theirs). Then decide: + --plan resolve per unit from the plan template written beside + the .mpr (.merge-plan.json): set "take" to + "ours"/"theirs" per unit; the .mpr index is repaired + from the mixed files. Modified-on-both units only — + added/deleted/moved conflicts go to Studio Pro. + --ours / --theirs take one side of every conflicted model file + (deterministic; mx is not consulted again) + Studio Pro (Version Control › Merge Changes Here) for mixed + within-unit resolutions — still the tool when a decision + must mix both sides inside one unit; - a conflict in non-model files (text sources) is left in place for manual resolution, like plain git. @@ -72,6 +81,7 @@ func init() { gitMergeCmd.Flags().StringP("message", "m", "", "merge commit message (default: git's)") gitMergeCmd.Flags().Bool("ours", false, "on a model conflict mx cannot resolve, take our side of every conflicted model file") gitMergeCmd.Flags().Bool("theirs", false, "on a model conflict mx cannot resolve, take their side of every conflicted model file") + gitMergeCmd.Flags().String("plan", "", "resolve per unit from a plan file written by a previous run (.merge-plan.json)") gitMergeCmd.Flags().Bool("push", false, "push the branch and the notes ref to their remote afterwards") gitCmd.AddCommand(gitMergeCmd) } @@ -81,23 +91,31 @@ func runGitMerge(cmd *cobra.Command, args []string) error { message, _ := cmd.Flags().GetString("message") ours, _ := cmd.Flags().GetBool("ours") theirs, _ := cmd.Flags().GetBool("theirs") + plan, _ := cmd.Flags().GetString("plan") push, _ := cmd.Flags().GetBool("push") project, _ := cmd.Flags().GetString("project") strategy := "" switch { case ours && theirs: return errors.New("--ours and --theirs cannot be combined") + case ours && plan != "", theirs && plan != "": + return errors.New("--plan cannot be combined with --ours/--theirs: the plan is the finer decision") case ours: strategy = "ours" case theirs: strategy = "theirs" } - return mergeRun(project, ref, message, strategy, push, realMxGitMerge, cmd.OutOrStdout()) + return mergeRun(project, ref, message, strategy, plan, push, mxRunners{merge: realMxGitMerge, diff: realMxDiff}, cmd.OutOrStdout()) } -// mxMergeRunner runs `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT -// THEIRS_COMMIT` and returns mx's exit code and output. +// mxRunners are the mx invocations, injected for tests. type mxMergeRunner func(base, mine, theirs string, shas [3]string) (int, string, error) +type mxDiffRunner func(base, mine, outJSON string) error + +type mxRunners struct { + merge mxMergeRunner + diff mxDiffRunner +} // mergeRun is the command without its wiring. runMx is injected for tests. // mergeRun is the command without its wiring. runMx is injected for tests. @@ -112,7 +130,7 @@ type mxMergeRunner func(base, mine, theirs string, shas [3]string) (int, string, // with no strategy the merge is aborted and both sides' changes are // printed for review; `--ours`/`--theirs` then take one side wholesale // (mx is not consulted — the human has decided). -func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRunner, out io.Writer) error { +func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners mxRunners, out io.Writer) error { switch strategy { case "", "ours", "theirs": default: @@ -139,6 +157,10 @@ func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRu return errors.New("detached HEAD: check out a branch before merging") } + if plan != "" { + return applyPlan(root, repo, abs, mprPath, ref, plan, message, branch, push, out) + } + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) if mergeErr == nil { if strings.Contains(mergeOut, "Already up to date") { @@ -221,7 +243,7 @@ func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRu version := projectMendixVersion(mprPath) fmt.Fprintf(out, "Model files conflict (%s); merging the model with mx %s…\n", strings.Join(shortNames(model), ", "), version) - code, mxOut, runErr := runMx( + code, mxOut, runErr := runners.merge( filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), abs, filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), @@ -235,15 +257,29 @@ func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRu return fmt.Errorf("mx git-merge failed: %w", runErr) } if code != 0 { - // The review step: show what each side changed, then leave nothing - // behind. The human decides — --ours/--theirs or Studio Pro. + // The review step: what each side changed, at property level (mx + // diff), plus a per-unit plan template. Then leave nothing behind — + // the human decides. + if runners.diff != nil { + oursJSON := filepath.Join(tmp, "ours-diff.json") + theirsJSON := filepath.Join(tmp, "theirs-diff.json") + d1 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), abs, oursJSON) + d2 := runners.diff(filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), theirsJSON) + if d1 == nil && d2 == nil { + fmt.Fprintln(out, propertyReport(oursJSON, theirsJSON, conflictedUnitIDs(model))) + } + } + if err := writePlanTemplate(mprPath, baseSha, oursSha, theirsSha, model); err != nil { + fmt.Fprintf(out, " (could not write the plan template: %v)\n", err) + } else { + fmt.Fprintf(out, "Plan template: %s — set \"take\" per unit, re-run with --plan.\n", filepath.Base(mprPath)+".merge-plan.json") + } cleanWorktrees() - printConflictReport(root, repo, baseSha, oursSha, theirsSha, out) if _, abortErr := gitOutIn(root, nil, "merge", "--abort"); abortErr != nil { return fmt.Errorf("the model merge has conflicts and aborting it failed (%v); resolve in Studio Pro", abortErr) } fmt.Fprintln(out, "Merge aborted; the working tree is back to the pre-merge state.") - return errors.New("the model merge needs a decision: re-run with --ours or --theirs to take one side wholesale, or resolve in Studio Pro (Version Control › Merge Changes Here)") + return errors.New("the model merge needs a decision: re-run with --plan to resolve per unit, --ours/--theirs to take one side wholesale, or resolve in Studio Pro (Version Control › Merge Changes Here)") } args := []string{"add", "--"} @@ -260,57 +296,6 @@ func mergeRun(mprPath, ref, message, strategy string, push bool, runMx mxMergeRu return nil } -// printConflictReport shows what each side changed — the review material a -// human needs before deciding how the conflict is resolved. -func printConflictReport(root string, repo *gitNoteRepo, baseSha, oursSha, theirsSha string, out io.Writer) { - base, err := repo.snapshot(baseSha) - if err != nil { - fmt.Fprintf(out, " (could not read the merge base: %v)\n", err) - return - } - ours, err := repo.snapshot(oursSha) - if err != nil { - fmt.Fprintf(out, " (could not read our side: %v)\n", err) - return - } - theirs, err := repo.snapshot(theirsSha) - if err != nil { - fmt.Fprintf(out, " (could not read their side: %v)\n", err) - return - } - oursChanges := computeModelChanges(base, ours, nil) - theirsChanges := computeModelChanges(base, theirs, nil) - theirsIDs := map[string]bool{} - for _, c := range theirsChanges { - theirsIDs[c.UnitID] = true - } - oursIDs := map[string]bool{} - for _, c := range oursChanges { - oursIDs[c.UnitID] = true - } - fmt.Fprintf(out, "\nBoth sides changed — a decision is needed on:\n") - for _, c := range oursChanges { - if theirsIDs[c.UnitID] { - fmt.Fprintf(out, " %s %s [%s] (unit %s)\n", c.UnitType, c.UnitName, c.Module, shortCommit(c.UnitID)) - } - } - fmt.Fprintf(out, "\nYours (%s):\n", shortCommit(oursSha)) - for _, c := range oursChanges { - fmt.Fprintf(out, " %-9s %s %s [%s]%s\n", c.Status, c.UnitType, c.UnitName, c.Module, oursTheirsMark(theirsIDs, c.UnitID)) - } - fmt.Fprintf(out, "\nTheirs (%s):\n", shortCommit(theirsSha)) - for _, c := range theirsChanges { - fmt.Fprintf(out, " %-9s %s %s [%s]%s\n", c.Status, c.UnitType, c.UnitName, c.Module, oursTheirsMark(oursIDs, c.UnitID)) - } -} - -func oursTheirsMark(other map[string]bool, unitID string) string { - if other[unitID] { - return " ◄ also changed on the other side" - } - return "" -} - // commitTheMerge commits the resolved merge (MERGE_HEAD is present), writes // the note and pushes. stagedPaths, when given, are staged first. func commitTheMerge(root string, repo *gitNoteRepo, stagedPaths []string, message, branch string, push bool, out io.Writer) error { @@ -414,3 +399,253 @@ func shortNames(paths []string) []string { } return out } + +// conflictedUnitIDs extracts the unit ids of conflicted .mxunit paths. +func conflictedUnitIDs(model []string) []string { + var ids []string + for _, p := range model { + if strings.HasSuffix(p, ".mxunit") { + ids = append(ids, strings.TrimSuffix(filepath.Base(p), ".mxunit")) + } + } + return ids +} + +// --- property-level conflict report and the per-unit plan ------------------- + +type mxDiffChange struct { + Change string `json:"change"` + ChangeDescription string `json:"changeDescription"` + Property string `json:"property"` + BaseValue string `json:"baseValue"` + MineValue string `json:"mineValue"` + ChangeDetails []mxDiffChange `json:"changeDetails"` +} + +type mxDiffUnit struct { + ID string `json:"id"` + Type string `json:"type"` + Change string `json:"change"` + ChangeDetails []mxDiffChange `json:"changeDetails"` +} + +type mxDiffFile struct { + UnitDifferences []mxDiffUnit `json:"unitDifferences"` +} + +// propertyReport renders the two mx diff files as one property-level report +// over the conflicted units: what each side changed. +func propertyReport(oursJSON, theirsJSON string, unitIDs []string) string { + want := map[string]bool{} + for _, id := range unitIDs { + want[id] = true + } + lines := map[string][]string{} + for _, f := range []struct{ side, path string }{{"Yours", oursJSON}, {"Theirs", theirsJSON}} { + raw, err := os.ReadFile(f.path) + if err != nil { + lines[f.side] = append(lines[f.side], fmt.Sprintf(" (diff unavailable: %v)", err)) + continue + } + var d mxDiffFile + if err := json.Unmarshal(bytes.TrimPrefix(raw, []byte("\xef\xbb\xbf")), &d); err != nil { + lines[f.side] = append(lines[f.side], fmt.Sprintf(" (diff unreadable: %v)", err)) + continue + } + var side []string + for _, u := range d.UnitDifferences { + if !want[u.ID] { + continue + } + side = append(side, flattenChanges(u.ChangeDetails, " ")...) + } + if len(side) == 0 { + side = append(side, " (no change details)") + } + lines[f.side] = side + } + var b strings.Builder + b.WriteString("\nProperty-level changes on the conflicted units:\n") + for _, side := range []string{"Yours", "Theirs"} { + fmt.Fprintf(&b, "%s:\n", side) + for _, l := range lines[side] { + b.WriteString(l) + b.WriteString("\n") + } + } + return b.String() +} + +// flattenChanges walks mx diff's change tree and yields one line per leaf: +// the description, and for property leaves the value movement. +func flattenChanges(changes []mxDiffChange, indent string) []string { + var out []string + for _, c := range changes { + if c.Property != "" { + out = append(out, fmt.Sprintf("%s%s: %s → %s", indent, c.Property, c.BaseValue, c.MineValue)) + continue + } + desc := c.ChangeDescription + if desc == "" { + desc = c.Change + } + out = append(out, fmt.Sprintf("%s%s", indent, desc)) + out = append(out, flattenChanges(c.ChangeDetails, indent+" ")...) + } + return out +} + +// mergePlanTemplate is the file a ManualResolutionRequired run writes beside +// the .mpr; the human sets "take" per unit and re-runs with --plan. +type mergePlanTemplate struct { + Base string `json:"base"` + Ours string `json:"ours"` + Theirs string `json:"theirs"` + Units []planUnit `json:"units"` +} + +type planUnit struct { + ID string `json:"id"` + Take string `json:"take"` +} + +func writePlanTemplate(mprPath, baseSha, oursSha, theirsSha string, model []string) error { + tmpl := mergePlanTemplate{Base: baseSha, Ours: oursSha, Theirs: theirsSha} + for _, id := range conflictedUnitIDs(model) { + tmpl.Units = append(tmpl.Units, planUnit{ID: id, Take: "ours"}) + } + b, err := json.MarshalIndent(tmpl, "", " ") + if err != nil { + return err + } + return os.WriteFile(mprPath+".merge-plan.json", b, 0o644) +} + +// applyPlan re-runs the merge (it must conflict the same way again), checks +// the plan against those revisions, resolves per unit, repairs the .mpr index +// from the mixed files and commits — with the note. +func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, message, branch string, push bool, out io.Writer) error { + raw, err := os.ReadFile(planPath) + if err != nil { + return fmt.Errorf("read plan: %w", err) + } + var plan struct { + Base string `json:"base"` + Ours string `json:"ours"` + Theirs string `json:"theirs"` + Units []planUnit `json:"units"` + } + if err := json.Unmarshal(raw, &plan); err != nil { + return fmt.Errorf("plan %s is not valid JSON: %w", planPath, err) + } + + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + if mergeErr == nil { + return errors.New("the merge no longer conflicts — the plan is unnecessary; nothing was done") + } + if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { + return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + } + baseSha := strings.TrimSpace(gitOutOr(root, "merge-base", "HEAD", "MERGE_HEAD")) + oursSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + theirsSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "MERGE_HEAD")) + if plan.Base != baseSha || plan.Ours != oursSha || plan.Theirs != theirsSha { + _, _ = gitOutIn(root, nil, "merge", "--abort") + return errors.New("stale plan: the merge revisions moved since it was written; delete it and re-run the merge to regenerate") + } + + conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) + var model []string + for _, p := range conflicted { + if isModelPath(repo, p) { + model = append(model, p) + } + } + + // Every planned unit must be modified on both sides; anything else + // (added on one side, deleted, moved) only Studio Pro can decide. + baseSnap, err := repo.snapshot(baseSha) + if err != nil { + return err + } + oursSnap, err := repo.snapshot(oursSha) + if err != nil { + return err + } + theirsSnap, err := repo.snapshot(theirsSha) + if err != nil { + return err + } + oursStatus := changeStatuses(computeModelChanges(baseSnap, oursSnap, nil)) + theirsStatus := changeStatuses(computeModelChanges(baseSnap, theirsSnap, nil)) + take := map[string]string{} + for _, u := range plan.Units { + if u.Take != "ours" && u.Take != "theirs" { + return fmt.Errorf("plan: unit %s has take %q (ours|theirs)", shortCommit(u.ID), u.Take) + } + if oursStatus[u.ID] != "Modified" || theirsStatus[u.ID] != "Modified" { + return fmt.Errorf("plan: unit %s is %s on ours and %s on theirs — only modified-on-both conflicts can be planned; resolve it in Studio Pro", shortCommit(u.ID), statusOrNone(oursStatus[u.ID]), statusOrNone(theirsStatus[u.ID])) + } + take[u.ID] = u.Take + } + + // Ours everywhere, then the planned units from theirs. + for _, p := range model { + if _, err := gitOutIn(root, nil, "checkout", "--ours", "--", p); err != nil { + return fmt.Errorf("take the ours version of %s: %w", p, err) + } + } + for _, id := range conflictedUnitIDs(model) { + if take[id] != "theirs" { + continue + } + p := path.Join(repo.Contents, id[:2], id[2:4], id+".mxunit") + if _, err := gitOutIn(root, nil, "checkout", "--theirs", "--", p); err != nil { + return fmt.Errorf("take the theirs version of %s: %w", p, err) + } + fmt.Fprintf(out, " %s: theirs\n", shortCommit(id)) + } + // The .mpr index is ours; the mixed unit files have different bytes, so + // the hashes are rewritten from the files — files are the truth. + if err := fixHashes(mprPath, true, out); err != nil { + if !errors.Is(err, errHashIssuesRemain) { + return err + } + return errors.New("refusing to commit: the ContentsHash index has MISSING or ORPHAN entries (see above)") + } + + args := []string{"add", "--"} + args = append(args, model...) + if _, err := gitOutIn(root, nil, args...); err != nil { + return fmt.Errorf("stage the resolved model files: %w", err) + } + return commitTheMerge(root, repo, nil, message, branch, push, out) +} + +func changeStatuses(changes []mxModelChange) map[string]string { + out := map[string]string{} + for _, c := range changes { + out[c.UnitID] = c.Status + } + return out +} + +func statusOrNone(s string) string { + if s == "" { + return "unchanged" + } + return s +} + +// realMxDiff shells out to `mx diff BASE MINE OUTPUT` (resolved like git-merge). +func realMxDiff(base, mine, outJSON string) error { + version := projectMendixVersion(mine) + mxPath, err := docker.ResolveMxForVersion("", version) + if err != nil { + return fmt.Errorf("no mx for Mendix %s: %w", version, err) + } + if b, err := exec.Command(mxPath, "diff", base, mine, outJSON).CombinedOutput(); err != nil { + return fmt.Errorf("mx diff: %v: %s", err, strings.TrimSpace(string(b))) + } + return nil +} diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index a727bf953..668feed6d 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -5,6 +5,7 @@ package main import ( "bytes" "encoding/json" + "fmt" "os" "path/filepath" "strings" @@ -49,6 +50,38 @@ func modelConflictRepo(t *testing.T) (string, string) { return dir, mprPath } +// twoUnitConflictRepo is a modelConflictRepo where BOTH the domain model and +// the microflow unit were changed on each side — two conflicted units, so a +// plan can take one unit from ours and the other from theirs. +func twoUnitConflictRepo(t *testing.T) (string, string) { + t.Helper() + dir, mprPath := doctorNoteRepo(t) + units := baseUnits(t) + units[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", mainSideKey)} + units[tMf] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Save", mainSideKey)} + writeProject(t, dir, "11.13.0", units) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "main side") + gitIn(t, dir, "switch", "-q", "-c", "side", "HEAD~1") + side := baseUnits(t) + side[tDM] = tUnit{tMod, unitDoc(t, "DomainModels$DomainModel", "", sideKey)} + side[tMf] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Save", sideKey)} + writeProject(t, dir, "11.13.0", side) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "side change") + gitIn(t, dir, "switch", "-q", "feature") + _, err := gitInE(t, dir, "merge", "side") + if err != nil && !strings.Contains(err.Error(), "exit status") { + t.Fatalf("fixture: expected a conflicted merge, got %v", err) + } + return dir, mprPath +} + +// unitRel is the mprcontents path of a unit id inside a work tree. +func unitRel(id string) string { + return "mprcontents/" + id[:2] + "/" + id[2:4] + "/" + id + ".mxunit" +} + var ( mainSideKey = bsonKeyValue("DocA", "main") sideKey = bsonKeyValue("DocB", "side") @@ -72,12 +105,12 @@ func TestGitMerge_CleanMergeTextFileWritesNote(t *testing.T) { gitIn(t, dir, "commit", "-q", "-m", "docs side") gitIn(t, dir, "switch", "-q", "feature") - noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { t.Fatal("mx must not run on a merge git resolves itself") return 0, "", nil - } + }} var out bytes.Buffer - if err := mergeRun(mprPath, "docs", "", "", false, noMx, &out); err != nil { + if err := mergeRun(mprPath, "docs", "", "", "", false, noMx, &out); err != nil { t.Fatalf("mergeRun: %v\n%s", err, out.String()) } parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) @@ -99,12 +132,12 @@ func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { var gotBase, gotMine, gotTheirs string var gotShas [3]string - runner := func(base, mine, theirs string, shas [3]string) (int, string, error) { + runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { gotBase, gotMine, gotTheirs, gotShas = base, mine, theirs, shas return 0, "Merge successful.", nil - } + }} var out bytes.Buffer - if err := mergeRun(mprPath, "side", "", "", false, runner, &out); err != nil { + if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { t.Fatalf("mergeRun: %v\n%s", err, out.String()) } if !strings.Contains(gotBase, "mx-git-lab") && !strings.Contains(filepath.Base(gotBase), ".mpr") { @@ -140,11 +173,11 @@ func TestGitMerge_ModelConflictUnresolvedAborts(t *testing.T) { doctorEnv(t, dir) before := commitSubject(t, dir) - fails := func(base, mine, theirs string, shas [3]string) (int, string, error) { + fails := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil - } + }} var out bytes.Buffer - err := mergeRun(mprPath, "side", "", "", false, fails, &out) + err := mergeRun(mprPath, "side", "", "", "", false, fails, &out) if err == nil || !strings.Contains(err.Error(), "Studio Pro") { t.Fatalf("want a Studio Pro referral, got %v", err) } @@ -181,12 +214,12 @@ func TestGitMerge_NonModelConflictLeftAlone(t *testing.T) { } calls := 0 - counting := func(base, mine, theirs string, shas [3]string) (int, string, error) { + counting := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { calls++ return 0, "", nil - } + }} var out bytes.Buffer - merr := mergeRun(mprPath, "side", "", "", false, counting, &out) + merr := mergeRun(mprPath, "side", "", "", "", false, counting, &out) if merr == nil { t.Fatal("a non-model conflict must be reported, not silently passed") } @@ -205,12 +238,12 @@ func TestGitMerge_AlreadyUpToDate(t *testing.T) { doctorEnv(t, dir) before := commitSubject(t, dir) - noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { t.Fatal("mx must not run") return 0, "", nil - } + }} var out bytes.Buffer - if err := mergeRun(mprPath, "HEAD~1", "", "", false, noMx, &out); err != nil { + if err := mergeRun(mprPath, "HEAD~1", "", "", "", false, noMx, &out); err != nil { t.Fatalf("mergeRun: %v", err) } if got := commitSubject(t, dir); got != before { @@ -229,16 +262,16 @@ func TestGitMerge_ConflictReportShowsBothSidesAndAborts(t *testing.T) { doctorEnv(t, dir) before := commitSubject(t, dir) - noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil - } + }} var out bytes.Buffer - err := mergeRun(mprPath, "side", "", "", false, noMx, &out) + err := mergeRun(mprPath, "side", "", "", "", false, noMx, &out) if err == nil || !strings.Contains(err.Error(), "--ours") { t.Fatalf("the report must point at the resolution flags, got %v", err) } report := out.String() - for _, want := range []string{"DomainModels$DomainModel", "also changed on the other side", "Yours", "Theirs"} { + for _, want := range []string{"Plan template", "merge-plan.json", "Merge aborted"} { if !strings.Contains(report, want) { t.Errorf("report must contain %q:\n%s", want, report) } @@ -260,12 +293,12 @@ func TestGitMerge_SideTakingResolves(t *testing.T) { dir, mprPath := modelConflictRepo(t) doctorEnv(t, dir) - noMx := func(base, mine, theirs string, shas [3]string) (int, string, error) { + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { t.Fatal("--ours must not consult mx") return 0, "", nil - } + }} var out bytes.Buffer - if err := mergeRun(mprPath, "side", "", "ours", false, noMx, &out); err != nil { + if err := mergeRun(mprPath, "side", "", "ours", "", false, noMx, &out); err != nil { t.Fatalf("mergeRun --ours: %v\n%s", err, out.String()) } parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) @@ -286,12 +319,12 @@ func TestGitMerge_SideTakingResolves(t *testing.T) { // --theirs flips to the other side, deterministically. dir2, mprPath2 := modelConflictRepo(t) - theirs := func(base, mine, theirs string, shas [3]string) (int, string, error) { + theirs := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { t.Fatal("--theirs must not consult mx") return 0, "", nil - } + }} var out2 bytes.Buffer - if err := mergeRun(mprPath2, "side", "", "theirs", false, theirs, &out2); err != nil { + if err := mergeRun(mprPath2, "side", "", "theirs", "", false, theirs, &out2); err != nil { t.Fatalf("mergeRun --theirs: %v\n%s", err, out2.String()) } page2 := filepath.Join(dir2, "mprcontents", tDM[:2], tDM[2:4], tDM+".mxunit") @@ -303,3 +336,112 @@ func TestGitMerge_SideTakingResolves(t *testing.T) { t.Error("--theirs must leave their side of the unit on disk") } } + +// A plan resolves per unit: tDM from ours, tMf from theirs — one merge commit, +// hashes repaired from the mixed files, note attached. +func TestGitMerge_PlanMixesPerUnit(t *testing.T) { + dir, mprPath := twoUnitConflictRepo(t) + doctorEnv(t, dir) + base := strings.TrimSpace(gitOutT(t, dir, "merge-base", "HEAD", "side")) + ours := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")) + theirs := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "side")) + plan := fmt.Sprintf(`{"base":%q,"ours":%q,"theirs":%q,"units":[{"id":%q,"take":"ours"},{"id":%q,"take":"theirs"}]}`, + base, ours, theirs, tDM, tMf) + planPath := filepath.Join(dir, "merge-plan.json") + if err := os.WriteFile(planPath, []byte(plan), 0o644); err != nil { + t.Fatal(err) + } + + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("--plan must not consult mx") + return 0, "", nil + }} + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", planPath, false, noMx, &out); err != nil { + t.Fatalf("mergeRun --plan: %v\n%s", err, out.String()) + } + assertUnitHas := func(id, key string, want bool) { + t.Helper() + raw, err := os.ReadFile(filepath.Join(dir, unitRel(id))) + if err != nil { + t.Fatal(err) + } + if got := bytes.Contains(raw, []byte(key)); got != want { + t.Errorf("%s contains %q = %v, want %v", id, key, got, want) + } + } + assertUnitHas(tDM, "DocA", true) + assertUnitHas(tDM, "DocB", false) + assertUnitHas(tMf, "DocB", true) + assertUnitHas(tMf, "DocA", false) + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } + var out2 bytes.Buffer + if err := fixHashes(mprPath, false, &out2); err != nil { + t.Errorf("hashes drift after the plan merge: %v\n%s", err, out2.String()) + } +} + +// A plan whose revisions no longer match the merge state is stale and refused. +func TestGitMerge_PlanStaleRefused(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + plan := `{"base":"0000000000000000000000000000000000000000","ours":"0000000000000000000000000000000000000001","theirs":"0000000000000000000000000000000000000002","units":[]}` + planPath := filepath.Join(dir, "merge-plan.json") + if err := os.WriteFile(planPath, []byte(plan), 0o644); err != nil { + t.Fatal(err) + } + noMx := mxRunners{} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", planPath, false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "stale") { + t.Fatalf("want a stale-plan refusal, got %v", err) + } + gitInE(t, dir, "merge", "--abort") // clean up the fixture +} + +// On ManualResolutionRequired the report is property-level (from mx diff) and +// a plan template is written beside the .mpr. +func TestGitMerge_ManualWritesPlanAndPropertyReport(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + fixture := `{"unitDifferences":[{"id":"` + tDM + `","type":"DomainModels$DomainModel","change":"Modified","changeDetails":[{"change":"Modified","changeDescription":"Attribute 'MyFirstModule.CreatedByMxcli.Nickname'","changeDetails":[{"property":"New type > Length","baseValue":"100","mineValue":"150"}]}]}]}` + mx := mxRunners{ + merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 2, `[{"ConflictResolutionDetails":"ManualResolutionRequired"}]`, nil + }, + diff: func(base, mine, outJSON string) error { + return os.WriteFile(outJSON, []byte(fixture), 0o644) + }, + } + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", "", false, mx, &out) + if err == nil { + t.Fatal("a manual conflict must end with the decision error") + } + report := out.String() + for _, want := range []string{"Length", "100", "150", "Nickname"} { + if !strings.Contains(report, want) { + t.Errorf("report must contain %q:\n%s", want, report) + } + } + planRaw, err := os.ReadFile(mprPath + ".merge-plan.json") + if err != nil { + t.Fatalf("plan template not written: %v", err) + } + var plan struct { + Units []struct { + Id string `json:"id"` + Take string `json:"take"` + } `json:"units"` + } + if err := json.Unmarshal(planRaw, &plan); err != nil { + t.Fatal(err) + } + if len(plan.Units) != 1 || plan.Units[0].Id != tDM || plan.Units[0].Take != "ours" { + t.Errorf("plan template: %+v", plan) + } + gitInE(t, dir, "merge", "--abort") // clean up the fixture +} diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index bc6c67299..fee22c505 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -235,11 +235,16 @@ the project's Mendix version): merge commit, with the note attached; - a conflict mx reports as `ManualResolutionRequired` is never merged silently: the merge is aborted, and **both sides' changes are printed for - review** — unit by unit, ours and theirs. Then decide: + review** — at property level, from `mx diff` ("New type > Length: + 100 → 150" on ours, "→ 200" on theirs). Then decide: + `--plan ` resolves **per unit** from the plan template written beside + the .mpr (`.merge-plan.json`): set `"take"` to `ours`/`theirs` per + unit, and the `.mpr` index is repaired from the mixed files (modified-on- + both units only; added/deleted/moved conflicts go to Studio Pro). `--ours` or `--theirs` takes one side of every conflicted model file - (deterministic; mx is not consulted again), or resolve for real in - Studio Pro (*Version Control › Merge Changes Here*), which stays the tool - for mixed resolutions; + (deterministic; mx is not consulted again). Resolutions that must mix both + sides *inside* one unit stay in Studio Pro (*Version Control › Merge + Changes Here*); - a conflict in non-model files (text sources) is left in place for manual resolution, like plain git. From 61c38e1dd0f7bf3568d6c028e37266b9186b7fbd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:39:26 +0000 Subject: [PATCH 29/44] fix(gitignore): ignore the unit-type index under testdata fixtures too The pattern `.mxcli/unit-types.idx` has a slash in the middle, so git anchors it to the repository root: the index mxcli writes next to a fixture a test opens in place (testdata/testapp-views/.mxcli/) showed up as untracked, though the comment above the line says it is meant to be ignored there. `**/` matches it at any depth: `git check-ignore` did not match the nested copy before, and matches it and the root one after. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 5b15d750c..0ae724663 100644 --- a/.gitignore +++ b/.gitignore @@ -73,4 +73,4 @@ snap-bson go-test-output.txt # MPR v2 unit-type index, written next to any project mxcli opens — including # the testdata/ fixtures tests open in place. -.mxcli/unit-types.idx +**/.mxcli/unit-types.idx From 4335dff9a133c4b2fbd5392b16f05fa84d329541 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:27:59 +0000 Subject: [PATCH 30/44] fix(layout): an error handler's row reserves the lane under its activity Without @position a custom error handler is a row 130px under its activity, and the main line carries on one spacing after the activity however long the row is. When the next activity had a handler too, its row started at its own x, inside the first: the first handler's end event and the second handler's first activity were both stored at 680;330 (ako/mxcli#1768, mendixlabs/mxcli#1672). The lane model from #1154 already made an activity with a handler wait for a reserved lane; the handler row was the occupant that never reserved it. addErrorHandlerFlow now reserves the lane to the row's measured right edge plus the activity gap, and measureStatements does the same and counts the row in its extent, so a branch or loop holding handler rows is measured as built. Without the measurer half, 12 of the new shapes put a merge on an activity. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...onsecutive-error-handler-rows-overlap.json | 1 + CHANGELOG.md | 1 + cmd/mxcli/syntax/features_microflow.go | 13 +- docs-site/src/language/microflow-structure.md | 14 +- .../1768-error-handler-rows-share-a-lane.mdl | 66 ++++ mdl/executor/cmd_microflows_builder_flows.go | 4 + mdl/executor/layout.go | 8 + mdl/executor/layout_handler_lane_1768_test.go | 337 ++++++++++++++++++ mdl/executor/layout_lanes.go | 34 ++ 9 files changed, 468 insertions(+), 10 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-29-consecutive-error-handler-rows-overlap.json create mode 100644 mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl create mode 100644 mdl/executor/layout_handler_lane_1768_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-consecutive-error-handler-rows-overlap.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-consecutive-error-handler-rows-overlap.json new file mode 100644 index 000000000..aff06834a --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-consecutive-error-handler-rows-overlap.json @@ -0,0 +1 @@ +{"area":"mdl/executor","date":"2026-10-29","symptom":"Two consecutive activities with custom error handlers and no @position drew their handler rows on top of each other: the first handler's end event and the second handler's first activity both at 680;330 ($First 360;200, $Body 520;200, $Second 680;200). layout flows --dry-run said 'already laid out' (same builder), MPR008 said nothing (end events not compared), mx check 0 errors. A @position on a later statement also moved every unplaced statement before it onto that statement.","cause":"addErrorHandlerFlow built the row 130px under the activity, running right, but never reserved the lower lane, while clearLowerLane already made the next activity with a handler wait for a reserved lane (the #1154 guard design); measureStatements likewise reserved only for guards and left handler rows out of the extent that fitContainerSize counts, so a loop with a long trailing row was built 110px wider than measured. Case 2: buildFlowGraph's start pre-scan took the first statement WITH an @position instead of the first statement.","file":"mdl/executor/layout_lanes.go (handlerRowWidth, reserveHandlerRow), mdl/executor/layout.go (measureStatements), mdl/executor/cmd_microflows_builder_flows.go (addErrorHandlerFlow), mdl/executor/cmd_microflows_builder_graph.go (start pre-scan), mdl/linter/rules/mpr008_overlapping_activities.go","insight":"The lane model already stated the rule ('an activity with an error handler waits for the lane'); one of the lane's two occupants never registered itself. Look for the occupant that does not reserve before inventing a mechanism. Reserve from the measurer, in builder and measurer alike: the builder-only half (control) left 12 shapes with a merge landing on an activity. Studio Pro reference: all 8 multi-object handlers in PedApp and ako/TestApp are COLUMNS straight under the activity (80-130px per step), none is a row and none has a consecutive pair; a column layout would make the collision impossible but moves every flow with a 2+ statement handler and needs a vertical layout for compound handler bodies, so the lane was kept (2 of 182 example flows move). Two older measurer gaps surfaced by the new invariant test and were left alone: an IF is measured 45px short (the split's centre is counted as its left edge in guardBranchInset and measureIfStatement), and a `join` is measured as an activity plus an end event although it draws nothing, so a handler row ending in joins reserves about twice its drawn width. Check overlaps with the whole node set: MPR008 skipped events, which hid exactly this overlap; on 292 Studio Pro flows the wider set adds 0 findings.","refs":["ako/mxcli#1768","mendixlabs/mxcli#1672"],"test":"TestSecondErrorHandlerRowClearsTheFirst, TestErrorHandlerRowsNeverOverlap, TestLoopAroundHandlerRowsIsMeasuredAsBuilt, TestStatementsBeforeAPlacedOneKeepTheirDerivedPlaces, TestOverlapComparesEvents"} diff --git a/CHANGELOG.md b/CHANGELOG.md index ea06bf20b..1d08a4181 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. +- **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 9cfd46949..ae96ed08c 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -932,11 +932,14 @@ func init() { "right and wraps onto a new row past 2880px; the line joining two rows leaves\n" + "the bottom of one and arrives on top of the next. A guard (if … return; end\n" + "if) drops its branch into the lane below and the main line carries straight\n" + - "on over it. A CASE of four or more branches leaves the split in three groups\n" + - "— top, right, bottom — so its lines do not cross. A statement that carries\n" + - "@position is never moved, and starts the row for what follows it. Prefer no\n" + - "@position at all to a few: hand-placed statements are not measured against\n" + - "what the builder puts around them. (#1154)\n\n" + + "on over it; a custom error handler's row hangs in that lane too, 130px under\n" + + "its activity. A statement that reaches into the lane — a decision, a loop, an\n" + + "activity with a handler of its own — waits for it to clear; a plain activity\n" + + "does not (ako/mxcli#1768). A CASE of four or more branches leaves the split\n" + + "in three groups — top, right, bottom — so its lines do not cross. A\n" + + "statement that carries @position is never moved, and starts the row for what\n" + + "follows it. Prefer no @position at all to a few: hand-placed statements are\n" + + "not measured against what the builder puts around them. (#1154)\n\n" + "@start and @merge position the two nodes that have no statement of their\n" + "own, so each is written on the statement it belongs to. Omit @start and the\n" + "start is placed one spacing unit left of the first activity, on its centre\n" + diff --git a/docs-site/src/language/microflow-structure.md b/docs-site/src/language/microflow-structure.md index d7494c2b5..71eb46801 100644 --- a/docs-site/src/language/microflow-structure.md +++ b/docs-site/src/language/microflow-structure.md @@ -152,11 +152,15 @@ $Order = CREATE Sales.Order (Status = 'New'); Positions are optional. A microflow written without them is laid out by mxcli: the main line runs left to right and wraps onto a new row once it passes two canvas widths (2880 px); a guard — `if … then …; return; end if` — drops its branch into the -lane below while the main line carries on above it; and a `case` of four or more -branches leaves the decision in three groups (top, right, bottom) so its lines do not -cross. A statement that carries `@position` is never moved, and becomes the start of -the row for the statements after it — so either place everything or nothing: a few -hand-placed statements are not measured against what is laid out around them. +lane below while the main line carries on above it; a custom error handler's row hangs +in that same lane, 130 px under its activity; and a `case` of four or more branches +leaves the decision in three groups (top, right, bottom) so its lines do not cross. +Whatever reaches down into the lane — a decision, a loop, an activity with an error +handler of its own — waits for the lane to clear, so two consecutive handlers never +share a row; a plain activity does not wait. A statement that carries `@position` is +never moved, and becomes the start of the row for the statements after it — so either +place everything or nothing: a few hand-placed statements are not measured against +what is laid out around them. To re-arrange a flow that already exists — one drawn in Studio Pro, or one whose positions no longer fit after edits — run `mxcli layout flows`. It applies this diff --git a/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl b/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl new file mode 100644 index 000000000..bf28982c1 --- /dev/null +++ b/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl @@ -0,0 +1,66 @@ +-- ============================================================================ +-- ako/mxcli#1768 (mendixlabs/mxcli#1672) — the error handlers of two +-- consecutive activities were drawn in one row, the second on the first +-- ============================================================================ +-- +-- Without @position a custom error handler is a row 130px under its activity, +-- running right, and the main line carries on one spacing after the activity +-- whatever the row's length. A second activity with a handler started its row +-- at its own x, inside the first. Reported, from `describe … with handles`: +-- +-- first handler: $FirstFailed 360;330, log 520;330, end event 680;330 +-- $Second 680;200, second handler: $SecondFailed 680;330 <- on the end event +-- +-- `layout flows --dry-run` called the flow already laid out, and MPR008 said +-- nothing, because it did not compare end events. +-- +-- Expected after exec: `describe microflow BugTest1768.SUB_TwoHandlers with +-- handles` keeps $First, its handler row and $Body where they were (360;200, +-- 360..680;330, 520;200) and places $Second at 790;200 with its row at +-- 790..1110;330 — clear of the first row's end, with the gap two activities +-- keep. `lint -r MPR008` reports nothing, and `layout flows --dry-run` calls +-- every flow below already laid out. + +mdl 1; + +create module BugTest1768; + +create non-persistent entity BugTest1768.Result (IsDone: Boolean, Code: Integer); + +create or modify microflow BugTest1768.SUB_TwoHandlers ($Url: String) returns BugTest1768.Result as $Result +begin + $First = call rest service get '{1}' with ({1} = $Url) (Timeout: 30) returns response on error without rollback begin + $FirstFailed = create BugTest1768.Result (Code = 1); + log error node 'BugTest1768' 'first failed'; + return $FirstFailed; + end error; + declare $Body String = $First/Content; + $Second = call rest service get '{1}' with ({1} = $Body) (Timeout: 30) returns response on error without rollback begin + $SecondFailed = create BugTest1768.Result (Code = 2); + log error node 'BugTest1768' 'second failed'; + return $SecondFailed; + end error; + $Done = create BugTest1768.Result (IsDone = true); + return $Done; +end; + +-- One more statement in the first handler: its row then reached 840;330, two +-- objects deep into the second row, and MPR008 reported only the one of the two +-- overlaps that was activity on activity. +create or modify microflow BugTest1768.SUB_TwoHandlersLonger ($Url: String) returns BugTest1768.Result as $Result +begin + $First = call rest service get '{1}' with ({1} = $Url) (Timeout: 30) returns response on error without rollback begin + $FirstFailed = create BugTest1768.Result (Code = 1); + log error node 'BugTest1768' 'first failed'; + log info node 'BugTest1768' 'first failed, details'; + return $FirstFailed; + end error; + declare $Body String = $First/Content; + $Second = call rest service get '{1}' with ({1} = $Body) (Timeout: 30) returns response on error without rollback begin + $SecondFailed = create BugTest1768.Result (Code = 2); + log error node 'BugTest1768' 'second failed'; + return $SecondFailed; + end error; + $Done = create BugTest1768.Result (IsDone = true); + return $Done; +end; diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index 00aa4dcb7..f8691d94d 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -1015,6 +1015,10 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in fb.returnEndIDs[id] = true } + // The row now occupies the lane under this activity's line, so the next + // statement that reaches into that lane waits for it (ako/mxcli#1768). + fb.reserveHandlerRow(sourceX, errorBody) + // If the error handler ends with RAISE ERROR or RETURN, it terminates there. // Otherwise, return the last activity ID so caller can create a merge. if errBuilder.endsWithReturn { diff --git a/mdl/executor/layout.go b/mdl/executor/layout.go index 1431cbe1f..55e2ec459 100644 --- a/mdl/executor/layout.go +++ b/mdl/executor/layout.go @@ -98,6 +98,14 @@ func (m *layoutMeasurer) measureStatements(stmts []ast.MicroflowStatement) Bound } else { cursor = left + bounds.Width } + if row := m.handlerRowWidth(stmt); row > 0 { + // The handler's row hangs in the same lane, from the activity's left + // edge, and the main line goes on above it — as reserveHandlerRow + // records it for the builder. + rowRight := left + row + laneBusy = max(laneBusy, rowRight+laneGap) + extent = max(extent, rowRight) + } extent = max(extent, cursor) prev = stmt } diff --git a/mdl/executor/layout_handler_lane_1768_test.go b/mdl/executor/layout_handler_lane_1768_test.go new file mode 100644 index 000000000..11d7a7313 --- /dev/null +++ b/mdl/executor/layout_handler_lane_1768_test.go @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// A custom error handler is drawn as a row under its activity, ErrorHandlerPitch +// below the main line. The main line does not wait for that row — the statement +// after the activity is one pitch further on, however long the row is — so when +// that next statement had a handler too, its row started inside the first one +// (ako/mxcli#1768, mendixlabs/mxcli#1672). Reported, from `describe … with +// handles`: the first handler's end event and the second handler's first +// activity both on 680;330. +const twoHandlersFlow = `create microflow MyModule.SUB_TwoHandlers ($Url: String) returns MyModule.Result as $Result +begin + $First = call rest service get '{1}' with ({1} = $Url) (Timeout: 30) returns response on error without rollback begin + $FirstFailed = create MyModule.Result (Code = 1); + log error node 'MyModule' 'first failed'; + return $FirstFailed; + end error; + declare $Body String = $First/Content; + $Second = call rest service get '{1}' with ({1} = $Body) (Timeout: 30) returns response on error without rollback begin + $SecondFailed = create MyModule.Result (Code = 2); + log error node 'MyModule' 'second failed'; + return $SecondFailed; + end error; + $Done = create MyModule.Result (IsDone = true); + return $Done; +end;` + +// canvasBoxOf is an object's box on its canvas: a Mendix position is the centre. +func canvasBoxOf(o microflows.MicroflowObject) canvasBox { + p, sz := o.GetPosition(), objectSize(o) + return canvasBox{p.X - sz.Width/2, p.Y - sz.Height/2, p.X + sz.Width/2, p.Y + sz.Height/2} +} + +func canvasLabel(o microflows.MicroflowObject) string { + p := o.GetPosition() + return fmt.Sprintf("%s %d;%d", typeShort(o), p.X, p.Y) +} + +// canvasOverlaps lists every pair of objects that share space on the same canvas. +// A loop's children are positioned in the loop's own frame, so each loop body is +// its own canvas, checked separately; notes are left out. +func canvasOverlaps(objects []microflows.MicroflowObject) []string { + var out []string + var plane []microflows.MicroflowObject + for _, o := range objects { + if _, ok := o.(*microflows.Annotation); ok { + continue + } + plane = append(plane, o) + if loop, ok := o.(*microflows.LoopedActivity); ok && loop.ObjectCollection != nil { + for _, inner := range canvasOverlaps(loop.ObjectCollection.Objects) { + out = append(out, "in loop: "+inner) + } + } + } + for i := range plane { + for j := i + 1; j < len(plane); j++ { + a, b := canvasBoxOf(plane[i]), canvasBoxOf(plane[j]) + if a.left < b.right && b.left < a.right && a.top < b.bottom && b.top < a.bottom { + out = append(out, canvasLabel(plane[i])+" overlaps "+canvasLabel(plane[j])) + } + } + } + return out +} + +func TestSecondErrorHandlerRowClearsTheFirst(t *testing.T) { + oc := buildLayout(t, twoHandlersFlow) + if overlaps := canvasOverlaps(oc.Objects); len(overlaps) > 0 { + t.Errorf("objects share a place:\n %s", strings.Join(overlaps, "\n ")) + } + + got := map[string]bool{} + for _, o := range oc.Objects { + got[canvasLabel(o)] = true + } + // Everything up to the second handler stays where it was: the first activity, + // its row, and the plain activity after it, which does not reach into the lane + // and so does not wait for it. + for _, want := range []string{ + "ActionActivity 360;200", // $First + "ActionActivity 360;330", // $FirstFailed + "ActionActivity 520;330", // log error 'first failed' + "EndEvent 680;330", // return $FirstFailed + "ActionActivity 520;200", // $Body + } { + if !got[want] { + t.Errorf("missing %s — a position that did not overlap anything moved", want) + } + } + // $Second reaches into the lane with its own row, so it waits for the first row + // to clear: that row ends at 690 (the end event's right edge), plus the 40px gap + // two activities keep, plus half an activity. The literal is the measurement, + // not the builder's constants, so it cannot agree with whatever they say. + for _, want := range []string{ + "ActionActivity 790;200", // $Second + "ActionActivity 790;330", // $SecondFailed, straight under its activity + } { + if !got[want] { + t.Errorf("missing %s", want) + } + } +} + +// handlerRuns are runs of activities whose error handler rows share a lane. The +// builder places a handler row the same way wherever the run is — on the main +// line, in a loop body, in either branch of a decision — so each is checked in +// all of them. +var handlerRuns = []struct { + name, body string + // rejoins: the handlers fall through to the main line instead of ending. + rejoins bool +}{ + {name: "two handlers with a plain activity between", body: ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + return; + end error; + log info node 'M' 'between'; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + log error node 'M' 'b2'; + return; + end error; + log info node 'M' 'done';`}, + {name: "the first handler longer", body: ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + log error node 'M' 'a3'; + return; + end error; + log info node 'M' 'between'; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + log error node 'M' 'b2'; + return; + end error; + log info node 'M' 'done';`}, + {name: "three handlers back to back", body: ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + return; + end error; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + log error node 'M' 'b2'; + return; + end error; + retrieve $C from M.Item on error without rollback begin + log error node 'M' 'c1'; + return; + end error; + log info node 'M' 'done';`}, + {name: "handlers that rejoin the main line", rejoins: true, body: ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + end error; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + log error node 'M' 'b2'; + end error; + log info node 'M' 'done';`}, +} + +// lanePartners put a handler row next to the lane's other occupants: a guard's +// branch, a decision's ELSE row, a handler row inside a decision's branch. +// +// They are checked where the builder places the statement after the decision +// itself — the main line and a loop body. Inside a branch the enclosing merge is +// placed from the measured width of that decision, and the measurer counts a +// split's x as its left edge, so every decision is measured 45px short whether +// or not a handler is involved: `if … else … end if; log …` in a THEN branch puts +// the closing merge 5px into that log. That is its own defect, not this one. +var lanePartners = []struct{ name, body string }{ + {"a guard after a handler", ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + return; + end error; + if $Flag then + log info node 'M' 'refused'; + return; + end if; + log info node 'M' 'done';`}, + {"a decision after a handler", ` + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + return; + end error; + if $Flag then + log info node 'M' 'yes'; + else + log info node 'M' 'no'; + end if; + log info node 'M' 'done';`}, + {"a handler after a guard", ` + if $Flag then + log info node 'M' 'refused'; + log info node 'M' 'refused, details'; + return; + end if; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + return; + end error; + log info node 'M' 'done';`}, + {"a handler row in a branch, a handler after the decision", ` + if $Flag then + retrieve $A from M.Item on error without rollback begin + log error node 'M' 'a1'; + log error node 'M' 'a2'; + log error node 'M' 'a3'; + return; + end error; + else + log info node 'M' 'no'; + end if; + retrieve $B from M.Item on error without rollback begin + log error node 'M' 'b1'; + return; + end error; + log info node 'M' 'done';`}, +} + +type shapeContext struct { + name string + wrap func(body string) string +} + +var ( + onMainLine = shapeContext{"main line", func(b string) string { return b }} + inLoopBody = shapeContext{"loop body", func(b string) string { + return "loop $It in $Items begin\n" + b + "\nend loop;\nlog info node 'M' 'after';" + }} + inThenBranch = shapeContext{"then branch", func(b string) string { + return "if $Other then\n" + b + "\nelse\n log info node 'M' 'else';\nend if;\nlog info node 'M' 'after';" + }} + inElseBranch = shapeContext{"else branch", func(b string) string { + return "if $Other then\n log info node 'M' 'then';\nelse\n" + b + "\nend if;\nlog info node 'M' 'after';" + }} + inThenWithoutElse = shapeContext{"then branch without else", func(b string) string { + return "if $Other then\n" + b + "\nend if;\nlog info node 'M' 'after';" + }} +) + +func handlerShapeFlow(body string) string { + return "create microflow M.F ($Flag: Boolean, $Other: Boolean, $Items: list of M.Item) begin\n" + body + "\nend;" +} + +// assertNoSharedPlace builds a flow and checks that no two of its objects share a +// place on any canvas. The builder places each statement and the measurer sizes +// the branch or loop it sits in; when the two disagree about where a statement +// went, a merge lands on it, which this catches too. +func assertNoSharedPlace(t *testing.T, src string, checkMergeDirection bool) { + t.Helper() + oc := buildLayout(t, src) + if overlaps := canvasOverlaps(oc.Objects); len(overlaps) > 0 { + t.Errorf("objects share a place:\n %s", strings.Join(overlaps, "\n ")) + } + if checkMergeDirection { + assertLayoutInvariants(t, t.Name(), oc) + } +} + +func TestErrorHandlerRowsNeverOverlap(t *testing.T) { + for _, run := range handlerRuns { + for _, ctx := range []shapeContext{onMainLine, inLoopBody, inThenBranch, inElseBranch, inThenWithoutElse} { + t.Run(run.name+" / "+ctx.name, func(t *testing.T) { + // A handler that falls through rejoins at a merge placed between its + // activity and the next one, left of a row of two or more: a + // backward edge, but not a shared place, and older than this test. + assertNoSharedPlace(t, handlerShapeFlow(ctx.wrap(run.body)), !run.rejoins) + }) + } + } + for _, shape := range lanePartners { + for _, ctx := range []shapeContext{onMainLine, inLoopBody} { + t.Run(shape.name+" / "+ctx.name, func(t *testing.T) { + assertNoSharedPlace(t, handlerShapeFlow(ctx.wrap(shape.body)), true) + }) + } + } +} + +// The measurer sizes a container before its body exists; the builder sizes the +// box from what it built, handler rows included. A loop around each run must come +// out exactly as wide as it was measured, or an IF holding the loop places its +// merge inside the box (the mendixlabs/mxcli#1598 class) — and once a handler row +// makes the next one wait, the measurer has to know about the wait as well as the +// builder. +func TestLoopAroundHandlerRowsIsMeasuredAsBuilt(t *testing.T) { + for _, run := range handlerRuns { + t.Run(run.name, func(t *testing.T) { + prog, errs := visitor.Build(handlerShapeFlow("loop $It in $Items begin\n" + run.body + "\nend loop;")) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + var loopStmt *ast.LoopStmt + for _, s := range mf.Body { + if l, ok := s.(*ast.LoopStmt); ok { + loopStmt = l + } + } + measured := (&layoutMeasurer{}).measureStatement(loopStmt) + + fb := &flowBuilder{posX: 200, posY: 200, spacing: HorizontalSpacing} + oc := fb.buildFlowGraph(mf.Body, mf.ReturnType) + var built model.Size + for _, o := range oc.Objects { + if l, ok := o.(*microflows.LoopedActivity); ok { + built = l.Size + } + } + if built.Width != measured.Width { + t.Errorf("loop built %dpx wide, measured %dpx", built.Width, measured.Width) + } + }) + } +} diff --git a/mdl/executor/layout_lanes.go b/mdl/executor/layout_lanes.go index 827efc98b..50edbdd59 100644 --- a/mdl/executor/layout_lanes.go +++ b/mdl/executor/layout_lanes.go @@ -13,6 +13,9 @@ // remembers how far it is taken. Only an element that reaches down into that lane — // another decision, a loop box, an activity with an error handler under it — has to // wait for it to clear; a plain activity does not. +// +// An error handler's row is the lane's other occupant: it hangs ErrorHandlerPitch +// under its activity and the main line carries on over it the same way. package executor import "github.com/mendixlabs/mxcli/mdl/ast" @@ -50,6 +53,37 @@ func reachesBelowMainLine(stmt ast.MicroflowStatement, bounds Bounds) bool { return len(getErrorHandlerBody(stmt)) > 0 } +// handlerRowWidth is how wide the row of a statement's custom error handler is, +// measured from the activity's left edge, or 0 when it has no handler body. The +// row's first element is centred under the activity (addErrorHandlerFlow), so +// the row starts where the activity does and runs right like a branch, end +// event included. +func (m *layoutMeasurer) handlerRowWidth(stmt ast.MicroflowStatement) int { + body := getErrorHandlerBody(stmt) + if len(body) == 0 { + return 0 + } + return m.measureBranch(body).Width +} + +// reserveHandlerRow marks the lane under the activity at activityX taken by its +// error handler's row (ako/mxcli#1768, mendixlabs/mxcli#1672). +// +// The row is laid out ErrorHandlerPitch under its activity and runs right, and the +// main line goes on one pitch after the activity regardless — which is right for a +// plain activity, whose row is above the handler's. But the next activity with a +// handler of its own started its row at its own x, inside this one: two consecutive +// REST calls each with a three-element handler put the first handler's end event +// and the second handler's first activity on the same point. clearLowerLane already +// made such a statement wait for a reserved lane; the row just never reserved it. +func (fb *flowBuilder) reserveHandlerRow(activityX int, body []ast.MicroflowStatement) { + if fb.measurer == nil || len(body) == 0 { + return + } + rowRight := activityX - ActivityWidth/2 + fb.measurer.measureBranch(body).Width + fb.reserveLowerLane(fb.posY, rowRight+laneGap) +} + // leftHalfOf is the distance from a statement's x to its left edge, as gapBetween // counts it. func leftHalfOf(stmt ast.MicroflowStatement) int { From 34d37878ca2922a62d23bfd9c0d4f254a2723063 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:28:19 +0000 Subject: [PATCH 31/44] fix(layout): a later @position no longer moves the statements before it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The start event goes one spacing left of the first statement when that statement carries @position. The pre-scan took the first statement that had one, wherever it was, and every unplaced statement before it follows the start: `declare $A …; @position(840, 200) declare $B …` stored both at 840;200 (ako/mxcli#1768 case 2, mendixlabs/mxcli#1672). Only the first statement moves the start now, as alter's patch (patchDiff.startEvent, declared[0]) and the syntax help already assumed. The help and the language reference now say that statements before a placed one keep their derived places. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- CHANGELOG.md | 1 + cmd/mxcli/syntax/features_microflow.go | 6 +- docs-site/src/language/microflow-structure.md | 8 ++- .../1768-error-handler-rows-share-a-lane.mdl | 16 ++++- mdl/executor/cmd_microflows_builder_graph.go | 19 ++++-- .../layout_position_later_1768_test.go | 68 +++++++++++++++++++ 6 files changed, 106 insertions(+), 12 deletions(-) create mode 100644 mdl/executor/layout_position_later_1768_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d08a4181..8812b9df0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. - **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. +- **A `@position` on a later statement no longer moves the statements before it onto it** (ako/mxcli#1768, mendixlabs/mxcli#1672) — `declare $A String = $Url; @position(840, 200) declare $B String = $A;` stored both declares at 840;200, and MPR008 reported the overlap. The start event is placed one spacing left of the first statement when that statement carries `@position`, but the builder took the first statement that had one, wherever it was, and the unplaced statements before it follow the start. Only the first statement moves the start now, as `alter`'s patch and the syntax help already assumed: the flow above starts at 200;200 with `$A` at 360;200 and `$B` at 840;200. The syntax help and the language reference now say what happens to the statements before a placed one. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index ae96ed08c..505b2bcde 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -938,8 +938,10 @@ func init() { "does not (ako/mxcli#1768). A CASE of four or more branches leaves the split\n" + "in three groups — top, right, bottom — so its lines do not cross. A\n" + "statement that carries @position is never moved, and starts the row for what\n" + - "follows it. Prefer no @position at all to a few: hand-placed statements are\n" + - "not measured against what the builder puts around them. (#1154)\n\n" + + "follows it; the statements BEFORE it keep the places the builder derives for\n" + + "them, and only a @position on the first statement moves the start event.\n" + + "Prefer no @position at all to a few: hand-placed statements are not measured\n" + + "against what the builder puts around them. (#1154)\n\n" + "@start and @merge position the two nodes that have no statement of their\n" + "own, so each is written on the statement it belongs to. Omit @start and the\n" + "start is placed one spacing unit left of the first activity, on its centre\n" + diff --git a/docs-site/src/language/microflow-structure.md b/docs-site/src/language/microflow-structure.md index 71eb46801..1eca953c2 100644 --- a/docs-site/src/language/microflow-structure.md +++ b/docs-site/src/language/microflow-structure.md @@ -158,9 +158,11 @@ leaves the decision in three groups (top, right, bottom) so its lines do not cro Whatever reaches down into the lane — a decision, a loop, an activity with an error handler of its own — waits for the lane to clear, so two consecutive handlers never share a row; a plain activity does not wait. A statement that carries `@position` is -never moved, and becomes the start of the row for the statements after it — so either -place everything or nothing: a few hand-placed statements are not measured against -what is laid out around them. +never moved, and becomes the start of the row for the statements after it; the +statements before it keep the places the builder derives for them, and only a +`@position` on the first statement moves the start event. So either place everything +or nothing: a few hand-placed statements are not measured against what is laid out +around them. To re-arrange a flow that already exists — one drawn in Studio Pro, or one whose positions no longer fit after edits — run `mxcli layout flows`. It applies this diff --git a/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl b/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl index bf28982c1..5d064f002 100644 --- a/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl +++ b/mdl-examples/bug-tests/1768-error-handler-rows-share-a-lane.mdl @@ -19,7 +19,13 @@ -- 360..680;330, 520;200) and places $Second at 790;200 with its row at -- 790..1110;330 — clear of the first row's end, with the gap two activities -- keep. `lint -r MPR008` reports nothing, and `layout flows --dry-run` calls --- every flow below already laid out. +-- every flow below already laid out except SUB_PositionedLater (layout flows +-- drops its @position). +-- +-- SUB_PositionedLater is the issue's second case: a @position on a later +-- statement used to move the start event to it and every unplaced statement +-- before it along with the start, so `declare $A` and `declare $B` were both +-- stored at 840;200. Expected now: start 200;200, $A 360;200, $B 840;200. mdl 1; @@ -64,3 +70,11 @@ begin $Done = create BugTest1768.Result (IsDone = true); return $Done; end; + +create or modify microflow BugTest1768.SUB_PositionedLater ($Url: String) returns String as $B +begin + declare $A String = $Url; + @position(840, 200) + declare $B String = $A; + return $B; +end; diff --git a/mdl/executor/cmd_microflows_builder_graph.go b/mdl/executor/cmd_microflows_builder_graph.go index 1a4ca5122..b8d681a81 100644 --- a/mdl/executor/cmd_microflows_builder_graph.go +++ b/mdl/executor/cmd_microflows_builder_graph.go @@ -36,15 +36,22 @@ func (fb *flowBuilder) buildFlowGraph(stmts []ast.MicroflowStatement, returns *a // Set baseY for branch restoration (this is the center line) fb.baseY = fb.posY - // Pre-scan: if the first statement carries an @position annotation, shift the - // StartEvent to be one spacing unit to the left of that position so it doesn't - // end up behind activities that use explicit coordinates. - for _, stmt := range stmts { - if ann := getStatementAnnotations(stmt); ann != nil && ann.Position != nil { + // If the first statement carries an @position annotation, shift the StartEvent + // to be one spacing unit to the left of that position so it doesn't end up + // behind activities that use explicit coordinates. + // + // The first statement only. This used to take the first statement that had an + // @position, wherever it was, so a position on a later statement moved the + // start — and every unplaced statement before it, which follows the start — to + // that statement's place: `declare $A …; @position(840, 200) declare $B …` + // stored both at 840;200 (ako/mxcli#1768). Statements before a placed one keep + // the places the builder derives for them, as alter's patch already assumed + // (patchDiff.startEvent reads declared[0]). + if len(stmts) > 0 { + if ann := getStatementAnnotations(stmts[0]); ann != nil && ann.Position != nil { fb.posX = ann.Position.X - fb.spacing fb.posY = ann.Position.Y fb.baseY = fb.posY - break } } diff --git a/mdl/executor/layout_position_later_1768_test.go b/mdl/executor/layout_position_later_1768_test.go new file mode 100644 index 000000000..1e3e9f7d5 --- /dev/null +++ b/mdl/executor/layout_position_later_1768_test.go @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// A statement placed with @position starts the row for what follows it, and the +// start event sits one spacing left of the first statement when THAT one is +// placed. The builder found "the first statement with an @position" instead of +// looking at the first statement, so a position written on a later statement +// moved the start event to it — and every unplaced statement before it along +// with the start, onto the placed one (ako/mxcli#1768 case 2, mendixlabs/mxcli#1672). +// Reported: `describe … with handles` printed @position(840, 200) for both +// `declare $A` and `declare $B`, and MPR008 reported the overlap. +func TestStatementsBeforeAPlacedOneKeepTheirDerivedPlaces(t *testing.T) { + oc := buildLayout(t, `create microflow MyModule.SUB_PositionedLater ($Url: String) returns String as $B +begin + declare $A String = $Url; + @position(840, 200) + declare $B String = $A; + return $B; +end;`) + if overlaps := canvasOverlaps(oc.Objects); len(overlaps) > 0 { + t.Errorf("objects share a place:\n %s", strings.Join(overlaps, "\n ")) + } + got := map[string]bool{} + for _, o := range oc.Objects { + got[canvasLabel(o)] = true + } + for _, want := range []string{ + "StartEvent 200;200", // where a flow with no placed first statement starts + "ActionActivity 360;200", // declare $A: one spacing after the start + "ActionActivity 840;200", // declare $B: where the script put it + "EndEvent 1000;200", // one spacing after $B + } { + if !got[want] { + t.Errorf("missing %s", want) + } + } +} + +// Control for the rule above: a placed FIRST statement still carries the start +// event with it, one spacing to its left on its centre line. +func TestPlacedFirstStatementStillCarriesTheStartEvent(t *testing.T) { + oc := buildLayout(t, `create microflow MyModule.SUB_PositionedFirst ($Url: String) returns String as $B +begin + @position(840, 300) + declare $A String = $Url; + declare $B String = $A; + return $B; +end;`) + got := map[string]bool{} + for _, o := range oc.Objects { + got[canvasLabel(o)] = true + } + for _, want := range []string{ + "StartEvent 680;300", + "ActionActivity 840;300", + "ActionActivity 1000;300", + } { + if !got[want] { + t.Errorf("missing %s", want) + } + } +} From 49dec833056678d54f07c5290d653f43363ef1cd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:28:31 +0000 Subject: [PATCH 32/44] fix(lint): MPR008 compares events and type splits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The overlap rule built its canvas from activities, loops, decisions and merges only, so an activity lying on an error handler's end event was never reported — the shape ako/mxcli#1768 reported, two handler rows on top of each other, passed MPR008 until an activity happened to land on an activity. Start, end, error, break and continue events and inheritance splits are on the canvas too, each judged with its stored size; an event that stores none is taken as Studio Pro's 20x20 rather than as an activity. On the 292 Studio Pro-authored flows of PedApp and ako/TestApp the wider set adds no finding. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- CHANGELOG.md | 1 + .../rules/mpr008_overlapping_activities.go | 26 +++++++++ .../mpr008_overlapping_activities_test.go | 57 +++++++++++++++++++ 3 files changed, 84 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8812b9df0..eafed1b56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. - **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. - **A `@position` on a later statement no longer moves the statements before it onto it** (ako/mxcli#1768, mendixlabs/mxcli#1672) — `declare $A String = $Url; @position(840, 200) declare $B String = $A;` stored both declares at 840;200, and MPR008 reported the overlap. The start event is placed one spacing left of the first statement when that statement carries `@position`, but the builder took the first statement that had one, wherever it was, and the unplaced statements before it follow the start. Only the first statement moves the start now, as `alter`'s patch and the syntax help already assumed: the flow above starts at 200;200 with `$A` at 360;200 and `$B` at 840;200. The syntax help and the language reference now say what happens to the statements before a placed one. +- **MPR008 reports an activity drawn on an event** (ako/mxcli#1768, mendixlabs/mxcli#1672) — the overlap rule compared activities, loops, decisions and merges only, so an error handler's activity lying on another handler's end event at 680;330 passed, and with one more statement in that handler it reported one of the two overlaps, the one that happened to be activity on activity. Start, end, error, break and continue events and type splits are compared too, each with its stored size (an event that stores none is taken as Studio Pro's 20×20, not as an activity). On the 292 flows of the Studio Pro-authored PedApp and ako/TestApp fixtures, and on 182 flows built from the `02*` and `06` doctype examples, the wider rule adds no finding. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/mdl/linter/rules/mpr008_overlapping_activities.go b/mdl/linter/rules/mpr008_overlapping_activities.go index 7b0f7152c..3e3da4ddd 100644 --- a/mdl/linter/rules/mpr008_overlapping_activities.go +++ b/mdl/linter/rules/mpr008_overlapping_activities.go @@ -15,6 +15,11 @@ import ( const activityBoxWidth = 120 const activityBoxHeight = 60 +// eventSize is the side of an event's box (start, end, error, break, continue) +// for one that stores no size: Studio Pro stores 20x20 on every event it draws +// (all 712 in ako/TestApp and PedApp), and so does mxcli. +const eventSize = 20 + // OverlappingActivitiesRule flags microflow activities whose canvas positions overlap. // // The most common cause is writing multiple MDL statements after a single @position @@ -124,6 +129,11 @@ func boxesOverlap(a, b actInfo) bool { func newActInfo(obj microflows.MicroflowObject, caption string) actInfo { p := obj.GetPosition() w, h := activityBoxWidth, activityBoxHeight + switch obj.(type) { + case *microflows.StartEvent, *microflows.EndEvent, *microflows.ErrorEvent, + *microflows.BreakEvent, *microflows.ContinueEvent: + w, h = eventSize, eventSize + } if s, ok := obj.(interface{ GetSize() model.Size }); ok { if sz := s.GetSize(); sz.Width > 0 && sz.Height > 0 { w, h = sz.Width, sz.Height @@ -170,8 +180,24 @@ func overlapPlanes(objects []microflows.MicroflowObject) [][]actInfo { } case *microflows.ExclusiveSplit: plane = append(plane, newActInfo(act, act.Caption)) + case *microflows.InheritanceSplit: + plane = append(plane, newActInfo(act, act.Caption)) case *microflows.ExclusiveMerge: plane = append(plane, newActInfo(act, "(merge)")) + // Events are nodes on the canvas too. Leaving them out hid an + // activity drawn on a handler's end event: two error handler rows + // laid on top of each other were reported only when an activity + // happened to land on an activity (ako/mxcli#1768). + case *microflows.StartEvent: + plane = append(plane, newActInfo(act, "(start)")) + case *microflows.EndEvent: + plane = append(plane, newActInfo(act, "(end)")) + case *microflows.ErrorEvent: + plane = append(plane, newActInfo(act, "(error)")) + case *microflows.BreakEvent: + plane = append(plane, newActInfo(act, "(break)")) + case *microflows.ContinueEvent: + plane = append(plane, newActInfo(act, "(continue)")) } } planes = append(planes, plane) diff --git a/mdl/linter/rules/mpr008_overlapping_activities_test.go b/mdl/linter/rules/mpr008_overlapping_activities_test.go index 20df3a146..3d9751308 100644 --- a/mdl/linter/rules/mpr008_overlapping_activities_test.go +++ b/mdl/linter/rules/mpr008_overlapping_activities_test.go @@ -147,3 +147,60 @@ func TestOverlapUsesStoredSizes(t *testing.T) { }) } } + +// ako/mxcli#1768: two error handler rows laid on top of each other put the first +// handler's end event and the second handler's first activity both on 680;330. +// MPR008 compared activities, splits and merges only, so it reported nothing — +// and with one more statement in the first handler it reported one of the two +// overlaps, the one that happened to be activity on activity. Events are nodes +// on the canvas and are compared too, with their own 20x20 box. +func TestOverlapComparesEvents(t *testing.T) { + act := func(x, y int) *microflows.ActionActivity { + a := µflows.ActionActivity{} + a.Position = model.Point{X: x, Y: y} + a.Size = model.Size{Width: 120, Height: 60} + return a + } + end := func(x, y int, sized bool) *microflows.EndEvent { + e := µflows.EndEvent{} + e.Position = model.Point{X: x, Y: y} + if sized { + e.Size = model.Size{Width: 20, Height: 20} + } + return e + } + for _, tc := range []struct { + name string + objs []microflows.MicroflowObject + want bool + }{ + {"end event on an activity (the report)", []microflows.MicroflowObject{end(680, 330, true), act(680, 330)}, true}, + {"end event one pitch after an activity", []microflows.MicroflowObject{act(520, 330), end(680, 330, true)}, false}, + // An event that stores no size is an event, not an activity: judged as a + // 120x60 box it would overlap an activity 80px away. + {"unsized end event 80px from an activity", []microflows.MicroflowObject{act(600, 200), end(680, 200, false)}, false}, + {"unsized end event inside an activity", []microflows.MicroflowObject{act(600, 200), end(640, 200, false)}, true}, + } { + t.Run(tc.name, func(t *testing.T) { + p := overlapPlanes(tc.objs)[0] + if len(p) != 2 { + t.Fatalf("got %d nodes on the canvas, want 2 — the event was left out", len(p)) + } + if got := boxesOverlap(p[0], p[1]); got != tc.want { + t.Errorf("overlap = %v, want %v (%+v, %+v)", got, tc.want, p[0], p[1]) + } + }) + } + + // Every kind of event, and a type split, is on the canvas. + nodes := []microflows.MicroflowObject{ + µflows.StartEvent{}, µflows.EndEvent{}, µflows.ErrorEvent{}, + µflows.BreakEvent{}, µflows.ContinueEvent{}, µflows.InheritanceSplit{}, + } + for i, n := range nodes { + n.SetPosition(model.Point{X: 100 + 200*i, Y: 200}) + } + if got := len(overlapPlanes(nodes)[0]); got != len(nodes) { + t.Errorf("%d of %d nodes are on the canvas", got, len(nodes)) + } +} From 83f22ae7cacaa6b7ec9f209439c234c14261b28d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:16:47 +0000 Subject: [PATCH 33/44] fix(executor): every nested flow body keeps the splice's member guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom (ako/mxcli#1769, mendixlabs/mxcli#1673, Mendix 11.12.4): `alter microflow … { replace loop … with begin … end; }` wrote `$X = find $List by Code = …` in the loop's body as Microflows$FindByExpression with the bare text `Code = …` when the fragment did not know $List's entity. check --references and exec reported nothing; mx check failed CE0117. The same find inserted on the main line was refused (ako/mxcli#891). Cause: confirmed. #891's qualifiedMembersOnly is set on the fragment's builder, but the loop body, the while body and the error-handler body are each built by a fresh flowBuilder whose fields the site listed by hand, and none listed the flag. So every member-bearing statement nested in a fragment (find, filter, sort, aggregate, change, at any depth) was written bare. `replace body of loop` went through the same loop builder and had the same hole. Fix: one construction helper, flowBuilder.nestedBuilder, for all three sites; it inherits the flag with the rest of the shared state. The error handler still sets its return type and label table itself, as before. TestNestedBuilder_InheritsSharedState classifies every flowBuilder field as inherited or the body's own and checks the helper against it by reflection, so the next field cannot be left out silently. With this alone the reported alter is refused, with nothing written, because the fragment does not type a list retrieved over an association; the next commit types it. Control: with the flag left out of nestedBuilder, the nested-guard test reports every statement in every nested shape written instead of refused, the PedApp test reports exec writing the replaced loop and replace body, and the structural test names the missing field. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- mdl/executor/cmd_microflows_builder.go | 54 ++++++ .../cmd_microflows_builder_control.go | 70 +------ mdl/executor/cmd_microflows_builder_flows.go | 43 ++--- .../cmd_microflows_builder_nested_test.go | 172 ++++++++++++++++++ ...ow_splice_nested_guard_1769_pedapp_test.go | 124 +++++++++++++ mdl/executor/flow_splice_nested_guard_test.go | 131 +++++++++++++ 6 files changed, 502 insertions(+), 92 deletions(-) create mode 100644 mdl/executor/cmd_microflows_builder_nested_test.go create mode 100644 mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go create mode 100644 mdl/executor/flow_splice_nested_guard_test.go diff --git a/mdl/executor/cmd_microflows_builder.go b/mdl/executor/cmd_microflows_builder.go index 4862606c1..8533b4d69 100644 --- a/mdl/executor/cmd_microflows_builder.go +++ b/mdl/executor/cmd_microflows_builder.go @@ -200,6 +200,60 @@ type flowBuilder struct { joinScopes map[string]flowBuilderVariableState } +// nestedBuilder returns the builder for a body nested in this one — a loop's, +// a while loop's, an error handler's — whose first activity is centred at +// (x, y). The body has its own position, graph and pending state, and +// inherits what says which flow is being built and how its statements are +// written: the variable scope, the backend and project lookups, the language, +// the notes, and the guards. Every nested body is built here, so a field is +// inherited by all of them or by none — bar what the one site that needs it +// sets (an error handler's return type and label table) — and +// TestNestedBuilder_InheritsSharedState fails until a new field is put on one +// side or the other. +// +// ako/mxcli#1769: each site listed the fields it copied, and none listed +// qualifiedMembersOnly, so a fragment spliced into a stored flow refused an +// unqualifiable member on its main line and wrote it in every body nested in +// it — `find $L by Code = …` in a replaced loop became a find by the invalid +// expression `Code = …` (CE0117). +func (fb *flowBuilder) nestedBuilder(x, y int) *flowBuilder { + return &flowBuilder{ + posX: x, + posY: y, + baseY: y, + spacing: HorizontalSpacing, + + varTypes: fb.varTypes, // one variable scope + declaredVars: fb.declaredVars, // (fixes a nil map panic) + textLang: fb.textLang, + measurer: fb.measurer, + backend: fb.backend, + quiet: fb.quiet, + hierarchy: fb.hierarchy, + restServices: fb.restServices, // for parameter classification + isNanoflow: fb.isNanoflow, + self: fb.self, + scriptFlows: fb.scriptFlows, + // The note registry, so a note declared outside the body and + // referenced on an activity in it attaches to the SAME Annotation + // rather than being refused. The describer emits exactly that (its + // label state is shared across the loop overlay), so a builder that + // refused it would reject its own DESCRIBE output. + // + // Across a loop boundary the result — Annotation in the parent + // collection, both AnnotationFlows in the parent collection, one + // destination inside the loop — was measured at 0 errors on mxbuild + // 11.13. mxbuild is the only oracle available here; Studio Pro is + // stricter in general, though this is the same split mxcli already + // ships for an ordinary loop-body note, whose flow is hoisted to the + // parent. (#1077) + annotationsByLabel: fb.annotationsByLabel, + storedNotes: fb.storedNotes, + // A body of a spliced fragment is spliced too (ako/mxcli#1769). + qualifiedMembersOnly: fb.qualifiedMembersOnly, + } +} + type flowBuilderVariableState struct { varTypes map[string]string declaredVars map[string]string diff --git a/mdl/executor/cmd_microflows_builder_control.go b/mdl/executor/cmd_microflows_builder_control.go index c5d9e66a7..73f65951e 100644 --- a/mdl/executor/cmd_microflows_builder_control.go +++ b/mdl/executor/cmd_microflows_builder_control.go @@ -711,38 +711,11 @@ func (fb *flowBuilder) addLoopStatement(s *ast.LoopStmt) model.ID { } } - // Build nested ObjectCollection for loop body - loopBuilder := &flowBuilder{ - textLang: fb.textLang, - posX: innerStartX, - posY: innerStartY, - baseY: innerStartY, - spacing: HorizontalSpacing, - varTypes: fb.varTypes, // Share variable scope - declaredVars: fb.declaredVars, // Share declared vars (fixes nil map panic) - measurer: fb.measurer, // Share measurer - backend: fb.backend, // Share backend - quiet: fb.quiet, - hierarchy: fb.hierarchy, // Share hierarchy - restServices: fb.restServices, // Share REST services for parameter classification - isNanoflow: fb.isNanoflow, - self: fb.self, - scriptFlows: fb.scriptFlows, - // Share the note registry, so a note declared outside the loop and - // referenced on a body activity attaches to the SAME Annotation rather - // than being refused. The describer emits exactly that (its label state - // is shared across the loop overlay), so a builder that refused it would - // reject its own DESCRIBE output. - // - // The resulting shape — Annotation in the parent collection, both - // AnnotationFlows in the parent collection, one destination inside the - // loop — was measured at 0 errors on mxbuild 11.13. mxbuild is the only - // oracle available here; Studio Pro is stricter in general, though this - // is the same split mxcli already ships for an ordinary loop-body note, - // whose flow is hoisted to the parent a few lines below. (#1077) - annotationsByLabel: fb.annotationsByLabel, - storedNotes: fb.storedNotes, - } + // Build nested ObjectCollection for loop body. It shares the variable + // scope, the note registry and the guards (nestedBuilder), and not the + // label table: the LoopedActivity owns a collection a sequence flow may + // not leave. + loopBuilder := fb.nestedBuilder(innerStartX, innerStartY) // Process loop body statements and connect them with flows. // pendingCase carries the deferred case value a merge-less split leaves for @@ -1092,37 +1065,8 @@ func (fb *flowBuilder) addWhileStatement(s *ast.WhileStmt) model.ID { loopLeftX = loopCenterX - loopWidth/2 } - loopBuilder := &flowBuilder{ - textLang: fb.textLang, - posX: innerStartX, - posY: innerStartY, - baseY: innerStartY, - spacing: HorizontalSpacing, - varTypes: fb.varTypes, - declaredVars: fb.declaredVars, - measurer: fb.measurer, - backend: fb.backend, - quiet: fb.quiet, - hierarchy: fb.hierarchy, - restServices: fb.restServices, - isNanoflow: fb.isNanoflow, - self: fb.self, - scriptFlows: fb.scriptFlows, - // Share the note registry, so a note declared outside the loop and - // referenced on a body activity attaches to the SAME Annotation rather - // than being refused. The describer emits exactly that (its label state - // is shared across the loop overlay), so a builder that refused it would - // reject its own DESCRIBE output. - // - // The resulting shape — Annotation in the parent collection, both - // AnnotationFlows in the parent collection, one destination inside the - // loop — was measured at 0 errors on mxbuild 11.13. mxbuild is the only - // oracle available here; Studio Pro is stricter in general, though this - // is the same split mxcli already ships for an ordinary loop-body note, - // whose flow is hoisted to the parent a few lines below. (#1077) - annotationsByLabel: fb.annotationsByLabel, - storedNotes: fb.storedNotes, - } + // A WHILE body is a loop body: same sharing as addLoopStatement's. + loopBuilder := fb.nestedBuilder(innerStartX, innerStartY) // Body bookkeeping is addLoopStatement's, verbatim: a WHILE body is a loop // body, so the same merge-less split needs the same handling. pendingCase diff --git a/mdl/executor/cmd_microflows_builder_flows.go b/mdl/executor/cmd_microflows_builder_flows.go index f8691d94d..2b8c23a15 100644 --- a/mdl/executor/cmd_microflows_builder_flows.go +++ b/mdl/executor/cmd_microflows_builder_flows.go @@ -867,35 +867,20 @@ func (fb *flowBuilder) addErrorHandlerFlow(sourceActivityID model.ID, sourceX in errorX := sourceX // Build error handler activities - errBuilder := &flowBuilder{ - textLang: fb.textLang, - posX: errorX, - posY: errorY, - baseY: errorY, - spacing: HorizontalSpacing, - returnType: fb.returnType, - varTypes: fb.varTypes, - declaredVars: fb.declaredVars, - measurer: fb.measurer, - backend: fb.backend, - quiet: fb.quiet, - hierarchy: fb.hierarchy, - restServices: fb.restServices, - isNanoflow: fb.isNanoflow, - self: fb.self, - scriptFlows: fb.scriptFlows, - // A handler's activities are merged into the PARENT's object collection - // below, so a note declared outside the handler and referenced inside it - // (or the reverse) lands in one collection — sharing the registry is - // sound here in a way it is not across a loop boundary (#1077). - annotationsByLabel: fb.annotationsByLabel, - storedNotes: fb.storedNotes, - // Same collection, so the same label table: `join recovered` inside a - // handler must find the `merge recovered` declared on the main path. - // This is the one rejoin MDL could not spell before, and the reason the - // registry is a shared pointer rather than per-builder state. - labelReg: fb.labels(), - } + // A handler's activities are merged into the PARENT's object collection + // below, so a note declared outside the handler and referenced inside it + // (or the reverse) lands in one collection — sharing the note registry + // (nestedBuilder) is sound here in a way it is not across a loop boundary + // (#1077). + errBuilder := fb.nestedBuilder(errorX, errorY) + // The handler keeps the flow's return type, which its pending-handler + // routing and the undeclared-`set` advice read; a loop body never had it. + errBuilder.returnType = fb.returnType + // Same collection, so the same label table: `join recovered` inside a + // handler must find the `merge recovered` declared on the main path. + // This is the one rejoin MDL could not spell before, and the reason the + // registry is a shared pointer rather than per-builder state. + errBuilder.labelReg = fb.labels() var lastErrID model.ID var lastErrCase string diff --git a/mdl/executor/cmd_microflows_builder_nested_test.go b/mdl/executor/cmd_microflows_builder_nested_test.go new file mode 100644 index 000000000..460f13947 --- /dev/null +++ b/mdl/executor/cmd_microflows_builder_nested_test.go @@ -0,0 +1,172 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "reflect" + "sort" + "testing" + "unsafe" + + "github.com/mendixlabs/mxcli/mdl/backend/mock" +) + +// nestedBuilderInherits classifies every field of flowBuilder: true when the +// builder of a nested body (a loop's, a while loop's, an error handler's) +// inherits it from its parent — nestedBuilder copies it — and false when the +// body keeps its own. A field added to flowBuilder fails +// TestNestedBuilder_InheritsSharedState until it is put here, which is the +// point: ako/mxcli#1769 was a field (qualifiedMembersOnly) that three +// hand-written copies each left out. +var nestedBuilderInherits = map[string]bool{ + // Which flow is built, and how its statements are written. + "varTypes": true, + "declaredVars": true, + "textLang": true, + "measurer": true, + "backend": true, + "quiet": true, + "hierarchy": true, + "restServices": true, + "isNanoflow": true, + "self": true, + "scriptFlows": true, + "annotationsByLabel": true, + "storedNotes": true, + "qualifiedMembersOnly": true, + + // Where the body is drawn: nestedBuilder's arguments, and the body's own + // lanes and rows (a loop box is sized to fit, so it never wraps). + "posX": false, "posY": false, "baseY": false, "spacing": false, + "lowerLane": false, "allowWrap": false, "row": false, + + // What the body builds, merged into the parent by the site. + "objects": false, "flows": false, "annotationFlows": false, "errors": false, + "statedNotes": false, "curveByOrigin": false, "rejoinMergeAt": false, + "returnEndIDs": false, "fallThroughEndID": false, "endsWithReturn": false, + "lastReturnEndID": false, "returnValue": false, + + // Per-statement pending state. + "nextConnectionPoint": false, "incomingRedirect": false, "nextFlowCase": false, + "nextFlowAnchor": false, "atBodyTail": false, "pendingAnnotations": false, + "previousStmtAnchor": false, "manualLoopBackTarget": false, "pendingJoin": false, + "emptyErrorHandlerFrom": false, "errorHandlerTailFrom": false, "errorHandlerSource": false, + "errorHandlerSkipVar": false, "errorHandlerTailCase": false, "errorHandlerTailAnchor": false, + "errorHandlerTailIsSource": false, "errorHandlerReturnValue": false, "pendingErrorHandlers": false, + + // Set by the one site that shares it: an error handler's body is in its + // parent's collection and on its path (addErrorHandlerFlow); a loop's is + // in a collection a sequence flow may not leave. + "returnType": false, "labelReg": false, + + // The whole flow's: the stored start event's position, and what + // buildFlowGraph collects from the body it builds. + "startPosition": false, "listInputVariables": false, "objectInputVariables": false, + "generatedVars": false, + + // Caches, filled on first use. + "assocCatalog": false, "microflowsCache": false, "microflowsCacheLoaded": false, + "nanoflowsCache": false, "nanoflowsCacheLoaded": false, + + // validateFlowBody's, which builds no nested builder. + "duplicateNamesOwnedElsewhere": false, "checkAssocShapes": false, + "assocObjectVars": false, "joinScopes": false, +} + +func TestNestedBuilder_InheritsSharedState(t *testing.T) { + typ := reflect.TypeOf(flowBuilder{}) + fields := map[string]bool{} + for i := 0; i < typ.NumField(); i++ { + name := typ.Field(i).Name + fields[name] = true + if _, ok := nestedBuilderInherits[name]; !ok { + t.Errorf("flowBuilder.%s is not classified: say in nestedBuilderInherits whether a nested body "+ + "inherits it (and copy it in nestedBuilder) or keeps its own", name) + } + } + var stale []string + for name := range nestedBuilderInherits { + if !fields[name] { + stale = append(stale, name) + } + } + sort.Strings(stale) + for _, name := range stale { + t.Errorf("nestedBuilderInherits names %s, which flowBuilder no longer has", name) + } + + parent := &flowBuilder{} + pv := reflect.ValueOf(parent).Elem() + for i := 0; i < pv.NumField(); i++ { + setNonZero(t, typ.Field(i).Name, settable(pv.Field(i))) + } + child := parent.nestedBuilder(11, 22) + cv := reflect.ValueOf(child).Elem() + placed := map[string]int64{"posX": 11, "posY": 22, "baseY": 22, "spacing": HorizontalSpacing} + for i := 0; i < cv.NumField(); i++ { + name := typ.Field(i).Name + p, c := settable(pv.Field(i)), settable(cv.Field(i)) + switch { + case placed[name] != 0: + if c.Int() != placed[name] { + t.Errorf("nestedBuilder(11, 22).%s = %d, want %d", name, c.Int(), placed[name]) + } + case nestedBuilderInherits[name]: + if !sameValue(p, c) { + t.Errorf("a nested body does not inherit flowBuilder.%s", name) + } + case !c.IsZero(): + t.Errorf("a nested body inherits flowBuilder.%s, classified as its own", name) + } + } +} + +// settable returns v — a field of an addressable struct, exported or not — +// as a value that can be read and set. +func settable(v reflect.Value) reflect.Value { + return reflect.NewAt(v.Type(), unsafe.Pointer(v.UnsafeAddr())).Elem() +} + +// setNonZero gives v a value distinguishable from its zero value and, for a +// reference, from any other instance of its type. +func setNonZero(t *testing.T, name string, v reflect.Value) { + t.Helper() + switch v.Kind() { + case reflect.Bool: + v.SetBool(true) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + v.SetInt(7) + case reflect.String: + v.SetString("x") + case reflect.Map: + v.Set(reflect.MakeMap(v.Type())) + case reflect.Slice: + v.Set(reflect.MakeSlice(v.Type(), 1, 1)) + case reflect.Pointer: + v.Set(reflect.New(v.Type().Elem())) + case reflect.Interface: + mb := reflect.ValueOf(&mock.MockBackend{}) + if !mb.Type().AssignableTo(v.Type()) { + t.Fatalf("flowBuilder.%s: no test value for %s", name, v.Type()) + } + v.Set(mb) + case reflect.Struct: + for i := 0; i < v.NumField(); i++ { + setNonZero(t, name, settable(v.Field(i))) + } + default: + t.Fatalf("flowBuilder.%s: no test value for kind %s", name, v.Kind()) + } +} + +// sameValue reports whether b is a's value — for a reference, the same +// instance, not an equal one. +func sameValue(a, b reflect.Value) bool { + switch a.Kind() { + case reflect.Map, reflect.Slice, reflect.Pointer: + return a.Pointer() == b.Pointer() + case reflect.Interface: + return !a.IsNil() && !b.IsNil() && a.Elem().Pointer() == b.Elem().Pointer() + } + return reflect.DeepEqual(a.Interface(), b.Interface()) +} diff --git a/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go new file mode 100644 index 000000000..4cc62bd1f --- /dev/null +++ b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// ako/mxcli#1769 (mendixlabs/mxcli#1673) on the Studio Pro-authored PedApp, +// through the real codec: the reported alter, which replaces a loop with its +// own text, wrote the find in its body as Microflows$FindByExpression `Code = +// $Lookup/Code` — check and exec silent, CE0117 from mx check — where `create +// microflow` writes Microflows$Find on MyModule.OrderLine.Code. + +const matchLinesSetup = `mdl 1; +create module MyModule; +create persistent entity MyModule.Order (Code: String(50)); +create persistent entity MyModule.OrderLine (Code: String(50), Status: String(20)); +create association MyModule.OrderLine_Order from MyModule.OrderLine to MyModule.Order; +create non-persistent entity MyModule.Lookup (Code: String(50)); +create microflow MyModule.SUB_MatchLines ($Order: MyModule.Order, $Lookups: List of MyModule.Lookup) returns Integer as $Found +begin + declare $Found Integer = 0; + retrieve $Lines from $Order/MyModule.OrderLine_Order; + $Open = filter $Lines by Status = 'open'; + loop $Lookup in $Lookups + begin + $Line = find $Lines by Code = $Lookup/Code; + if $Line != empty then + set $Found = $Found + 1; + end if; + end loop; + return $Found; +end;` + +// matchLinesFindIn returns, for each way an alter can write a find, the +// statement that writes one in list: the reported replace of the loop with +// its own text, a replace of its body, and plain inserts in the stored body +// and on the main line, which the guard of ako/mxcli#891 always covered. +func matchLinesFindIn(list string) map[string]string { + find := `$Line = find ` + list + ` by Code = $Lookup/Code;` + return map[string]string{ + "replace loop": `mdl 1; +alter microflow MyModule.SUB_MatchLines { + replace loop $Lookup in $Lookups with begin + loop $Lookup in $Lookups + begin + ` + find + ` + if $Line != empty then + set $Found = $Found + 1; + end if; + end loop; + end; +};`, + "replace body of loop": `mdl 1; +alter microflow MyModule.SUB_MatchLines { + replace body of loop $Lookup in $Lookups with begin + ` + find + ` + if $Line != empty then + set $Found = $Found + 1; + end if; + end; +};`, + "insert in the loop body": `mdl 1; +alter microflow MyModule.SUB_MatchLines { + insert after $Line begin $Again = find ` + list + ` by Code = $Lookup/Code; end; +};`, + "insert on the main line": `mdl 1; +alter microflow MyModule.SUB_MatchLines { + insert after $Open begin $First = find ` + list + ` by Code = 'x'; end; +};`, + } +} + +func openMatchLinesFixture(t *testing.T) (*Executor, *bytes.Buffer, string) { + t.Helper() + exec, out, dir := openPedAppCopy(t) + if err := agreeExec(t, exec, matchLinesSetup); err != nil { + t.Fatalf("setup: %v\n%s", err, out.String()) + } + return exec, out, dir +} + +// TestAlterMicroflow_FindOverUntypedListIsRefused: over a list whose entity +// the fragment does not know — the stored flow does not type a list +// operation's output — every alter is refused and writes nothing. On the main +// line and in a stored loop body it always was (ako/mxcli#891); in a loop the +// fragment brings, it was written as a find by expression. Check does not +// pass it silently: like the main-line refusal, it names the change as not +// predicted with exec's reason. +func TestAlterMicroflow_FindOverUntypedListIsRefused(t *testing.T) { + const want = `cannot qualify member "Code"` + for name, src := range matchLinesFindIn("$Open") { + t.Run(name, func(t *testing.T) { + exec, _, dir := openMatchLinesFixture(t) + before := projectFiles(t, dir) + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v\n%s", errs[0], src) + } + r := exec.CheckFlowVerdictsReport(prog) + var said []string + for _, u := range r.Unpredicted { + said = append(said, u.Reason) + } + for _, v := range r.Violations { + said = append(said, v.Message) + } + if !strings.Contains(strings.Join(said, "\n"), want) { + t.Errorf("check must name the refusal %q, said: %v", want, said) + } + if err := exec.ExecuteProgram(prog); err == nil || !strings.Contains(err.Error(), want) { + t.Errorf("exec must refuse with %q, got: %v", want, err) + } + if changed := diffProjectFiles(before, projectFiles(t, dir)); len(changed) != 0 { + t.Errorf("a refused alter wrote %v", changed) + } + }) + } +} diff --git a/mdl/executor/flow_splice_nested_guard_test.go b/mdl/executor/flow_splice_nested_guard_test.go new file mode 100644 index 000000000..f43add3be --- /dev/null +++ b/mdl/executor/flow_splice_nested_guard_test.go @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1769 (mendixlabs/mxcli#1673): a fragment spliced into a stored +// flow refuses a member it cannot qualify (qualifiedMembersOnly, #885) — but +// only on its own main line. Every body nested in it (a loop's, a while +// loop's, an error handler's) was built by a fresh flowBuilder that did not +// inherit the flag, so `find $L by Code = …` in a replaced loop over a list +// whose entity the fragment does not know was written as a find by the +// expression `Code = …`: check and exec silent, CE0117 from mx check. + +// nestedShapes wraps one statement in each body a fragment can nest it in. +// %s is the statement; $Cs is a typed list to iterate, $C a typed object. +var nestedShapes = map[string]string{ + "loop body": `loop $It in $Cs + begin + %s + end loop;`, + "while body": `while $C != empty + begin + %s + end while;`, + "error handler": `$H = call microflow Mod.GetItem() on error without rollback begin + %s + end error;`, + "loop in a loop": `loop $It in $Cs + begin + loop $Jt in $Cs + begin + %s + end loop; + end loop;`, + "error handler in a loop": `loop $It in $Cs + begin + $H = call microflow Mod.GetItem() on error without rollback begin + %s + end error; + end loop;`, + "while in an error handler": `$H = call microflow Mod.GetItem() on error without rollback begin + while $C != empty + begin + %s + end while; + end error;`, + "if in a loop": `loop $It in $Cs + begin + if $It != empty then + %s + end if; + end loop;`, +} + +// memberStatements are the statements that name a member of their +// variable's entity. %[1]s is a list variable, %[2]s an object variable. +var memberStatements = map[string]string{ + "find": `$F = find %[1]s by Name = 'x';`, + "filter": `$G = filter %[1]s by Name = 'x';`, + "sort": `$S = sort %[1]s by Name asc;`, + "aggregate": `$M = maximum %[1]s by Name;`, + "change": `change %[2]s (Name = 'x');`, +} + +func nestedFragmentBody(t *testing.T, shape, stmt string) []ast.MicroflowStatement { + t.Helper() + src := "mdl 1;\ncreate microflow Mod.Fragment ()\nbegin\n " + + strings.ReplaceAll(nestedShapes[shape], "%s", stmt) + "\nend;" + return parsedFlowBody(t, src) +} + +// TestSpliceFragment_NestedBodiesRefuseUnqualifiedMembers: every member a +// nested body names over a variable the fragment cannot type is refused, as +// it is on the fragment's main line. Control: over the typed $Cs / $C the +// same fragments build, and the find is a find by attribute. +func TestSpliceFragment_NestedBodiesRefuseUnqualifiedMembers(t *testing.T) { + ctx, a := spliceFixture(t) + for shape := range nestedShapes { + for name, stmt := range memberStatements { + untyped := strings.ReplaceAll(strings.ReplaceAll(stmt, "%[1]s", "$A"), "%[2]s", "$A") + _, err := a.buildFragment(ctx, nestedFragmentBody(t, shape, untyped)) + if err == nil || !strings.Contains(err.Error(), `cannot qualify member "Name"`) { + t.Errorf("%s in the %s over an untyped variable: want the refusal, got %v", name, shape, err) + } + + typed := strings.ReplaceAll(strings.ReplaceAll(stmt, "%[1]s", "$Cs"), "%[2]s", "$C") + frag, err := a.buildFragment(ctx, nestedFragmentBody(t, shape, typed)) + if err != nil { + t.Errorf("control: %s in the %s over a typed variable: %v", name, shape, err) + continue + } + if name != "find" { + continue + } + ops := fragmentListOperations(frag.Objects) + if len(ops) != 1 { + t.Errorf("control: find in the %s: want one list operation, got %d", shape, len(ops)) + continue + } + if f, ok := ops[0].(*microflows.FindByAttributeOperation); !ok || f.Attribute != "Mod.Item.Name" { + t.Errorf("control: find in the %s wrote %#v, want a find by attribute Mod.Item.Name", shape, ops[0]) + } + } + } +} + +// fragmentListOperations returns the list operations among objs, loop bodies +// included. +func fragmentListOperations(objs []microflows.MicroflowObject) []microflows.ListOperation { + var out []microflows.ListOperation + for _, obj := range objs { + switch o := obj.(type) { + case *microflows.ActionActivity: + if lo, ok := o.Action.(*microflows.ListOperationAction); ok { + out = append(out, lo.Operation) + } + case *microflows.LoopedActivity: + if o.ObjectCollection != nil { + out = append(out, fragmentListOperations(o.ObjectCollection.Objects)...) + } + } + } + return out +} From 8d3089e06efbdea83985bb96259f8d5f0270a907 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:37:53 +0000 Subject: [PATCH 34/44] fix(executor): type a stored retrieve over an association for a fragment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Symptom (ako/mxcli#1769, mendixlabs/mxcli#1673): after the previous commit, the reported alter — a replaced loop finding by member in $Lines, retrieved over an association before the loop — is refused with "cannot qualify member \"Code\"" where `create microflow` writes Microflows$Find on MyModule.OrderLine.Code. Cause: alterFlowContext.storedVariables, the variables a fragment starts from, typed parameters, declares, creates, database retrieves and calls, and left a retrieve over an association "Unknown": which end it yields depends on the end it starts from. A stored loop's iterator was not typed either, so a member of it in a fragment inserted in the loop body was refused too. Fix: storedVariables types both from the variable they start from, to a fixed point, so the order the stored flow lists them in does not matter. The retrieve's type is flowBuilder.associationRetrieveType: lookupAssociation, entityIsSubtypeOf and associationRetrieveYieldsList, the answer the full build and check --references already take. The parent is the FROM end and the child the TO end, so from the TO end (Order) a reference yields the list of FROM objects (OrderLine), from the FROM end the one TO object; a reference set is a list either way, an owner-Both reference one object either way. A self-reference, an unknown association, or a name two loops or retrieves give different origins stays untyped, and its member is refused rather than guessed. Controls: with storedVariables reverted, the association-typing tests and the PedApp by-attribute tests fail with the refusal; with the ambiguity guard off, the shared iterator name and the retrieve from it are typed. Measured on Mendix 11.13.0 (scratch app, real mx): before, FindByExpression "Code = $Lookup/Code" and CE0117; the previous commit alone, refused with exit 1 and the stored flow byte-identical; with both, Microflows$Find on MyModule.OrderLine.Code, mx check 0 errors, describe → exec "Unchanged microflow". Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...d-over-association-list-by-expression.json | 1 + CHANGELOG.md | 1 + ...-alter-loop-find-over-association-list.mdl | 61 ++++++ mdl/executor/cmd_alter_flow.go | 84 +++++++- .../cmd_microflows_builder_actions.go | 45 ++++ .../flow_splice_assoc_retrieve_test.go | 201 ++++++++++++++++++ ...ow_splice_nested_guard_1769_pedapp_test.go | 110 ++++++++++ 7 files changed, 501 insertions(+), 2 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-loop-find-over-association-list-by-expression.json create mode 100644 mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl create mode 100644 mdl/executor/flow_splice_assoc_retrieve_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-loop-find-over-association-list-by-expression.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-loop-find-over-association-list-by-expression.json new file mode 100644 index 000000000..266f7ebcb --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-alter-loop-find-over-association-list-by-expression.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-29", "symptom": "ako/mxcli#1769 (mendixlabs/mxcli#1673, 11.12.4): `alter microflow … { replace loop $Lookup in $Lookups with begin loop … begin $Line = find $Lines by Code = $Lookup/Code; … end loop; end; }`, $Lines retrieved over an association before the loop, stored the find as Microflows$FindByExpression `Code = $Lookup/Code`. check --references and exec reported nothing (\"Altered microflow\"), describe printed `find $Lines where Code = …`, mx check failed CE0117. create microflow writes Microflows$Find on MyModule.OrderLine.Code; the same find inserted on the main line was refused (#891's guard). `replace body of loop` had the same hole.", "cause": "Two. (1) #891's flowBuilder.qualifiedMembersOnly is set on the fragment's builder, but the loop, while and error-handler bodies are built by fresh flowBuilders whose fields each site listed by hand, and none listed the flag, so every member a nested body names (find/filter by member, sort, aggregate, change) at any depth was written bare. (2) alterFlowContext.storedVariables, the variables a fragment starts from, typed parameters, declares, creates, database retrieves and calls but left a retrieve over an association 'Unknown' (direction was the stated reason), so the member could not be qualified at all. Stored loop iterators were not typed either.", "file": "mdl/executor/cmd_microflows_builder.go, mdl/executor/cmd_microflows_builder_control.go, mdl/executor/cmd_microflows_builder_flows.go, mdl/executor/cmd_alter_flow.go, mdl/executor/cmd_microflows_builder_actions.go", "fix": "flowBuilder.nestedBuilder is the one constructor for loop, while and error-handler bodies and inherits qualifiedMembersOnly with the rest of the shared state (the handler still adds returnType and labelReg). storedVariables types a retrieve over an association from its start variable through flowBuilder.associationRetrieveType (lookupAssociation + entityIsSubtypeOf + associationRetrieveYieldsList, the answer the full build and check take: from the TO end the list of the FROM entity, from the FROM end the TO object, a reference set a list either way, owner Both one object) and a loop's iterator from its list, to a fixed point so candidate order does not matter. A self-reference, an unknown association, or a name two loops or retrieves give different origins stays untyped and its member is refused.", "test": "mdl/executor/cmd_microflows_builder_nested_test.go TestNestedBuilder_InheritsSharedState (every flowBuilder field classified inherited or own, checked against nestedBuilder by reflection); flow_splice_nested_guard_test.go TestSpliceFragment_NestedBodiesRefuseUnqualifiedMembers (find/filter/sort/aggregate/change in loop, while, handler, loop in loop, handler in loop, while in handler, if in loop; typed control); flow_splice_assoc_retrieve_test.go (both directions, reference set, one-to-one, chain out of order, iterator, self-reference, ambiguous iterator); flow_splice_nested_guard_1769_pedapp_test.go (PedApp, real codec: replace loop, replace body, insert in body, insert on main line all write Microflows$Find; over an untyped list all four refused with nothing written; describe->exec Unchanged). Controls: flag left out of nestedBuilder -> every nested shape written and the structural test names the field; storedVariables reverted -> the reported shape refused; ambiguity guard off -> $ItOrder typed. Platform 11.13.0: unfixed CE0117 (1 error); fix (1) alone refused, exit 1, stored flow byte-identical; fixed Microflows$Find on MyModule.OrderLine.Code, mx check 0 errors, describe->exec Unchanged.", "insight": "A guard set on a builder is only as good as the places that construct builders: grep `&flowBuilder{` before trusting a flowBuilder flag, and where a field is copied by hand at several sites, replace the copies with one constructor and a test that makes every field be classified. The reported statement was not the blast radius: every member-writing statement in every nested body, and `replace body of`, had it. Separately, an unqualifiable member is the symptom of an untyped variable; read what the fragment's starting scope types (storedVariables) before adding another refusal. Check reports a fragment-builder refusal as 'not predicted' with the reason, not MDL090, as it does #891's main-line refusal.", "ce": ["CE0117"], "refs": ["ako/mxcli#1769", "mendixlabs/mxcli#1673", "ako/mxcli#891", "ako/mxcli#885"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index eafed1b56..6390f9622 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. - **A `@position` on a later statement no longer moves the statements before it onto it** (ako/mxcli#1768, mendixlabs/mxcli#1672) — `declare $A String = $Url; @position(840, 200) declare $B String = $A;` stored both declares at 840;200, and MPR008 reported the overlap. The start event is placed one spacing left of the first statement when that statement carries `@position`, but the builder took the first statement that had one, wherever it was, and the unplaced statements before it follow the start. Only the first statement moves the start now, as `alter`'s patch and the syntax help already assumed: the flow above starts at 200;200 with `$A` at 360;200 and `$B` at 840;200. The syntax help and the language reference now say what happens to the statements before a placed one. - **MPR008 reports an activity drawn on an event** (ako/mxcli#1768, mendixlabs/mxcli#1672) — the overlap rule compared activities, loops, decisions and merges only, so an error handler's activity lying on another handler's end event at 680;330 passed, and with one more statement in that handler it reported one of the two overlaps, the one that happened to be activity on activity. Start, end, error, break and continue events and type splits are compared too, each with its stored size (an event that stores none is taken as Studio Pro's 20×20, not as an activity). On the 292 flows of the Studio Pro-authored PedApp and ako/TestApp fixtures, and on 182 flows built from the `02*` and `06` doctype examples, the wider rule adds no finding. +- **A find in a loop an `alter microflow` writes is stored as a find by attribute, not as a find by an invalid expression** (ako/mxcli#1769, mendixlabs/mxcli#1673) — `alter microflow … { replace loop … with begin … end; }` whose loop body held `$Line = find $Lines by Code = $Lookup/Code;`, with `$Lines` retrieved over an association before the loop, stored `Microflows$FindByExpression` with the bare text `Code = $Lookup/Code`: `check --references` and `exec` reported nothing, `describe` printed `find $Lines where …`, and mx check failed CE0117 "Error(s) in expression." `create microflow` stores the same find as `Microflows$Find` on `MyModule.OrderLine.Code`. Two causes. The guard that refuses a member a spliced fragment cannot qualify (ako/mxcli#891) applied on the fragment's main line only, not in a loop, while loop or error handler inside it, and so did not cover `replace body of loop` either: there any find, filter, sort, aggregate or change member was written bare. And the fragment did not know what a stored retrieve over an association holds, nor a stored loop's iterator. Both are now typed as `create microflow` types them: from the association's TO end a retrieve yields the list of its FROM entity, from the FROM end the one TO object. So the reported alter writes `Microflows$Find`, and what still cannot be typed (a self-reference, a list operation's output) is refused with nothing written, wherever the statement sits. Measured on Mendix 11.13.0: CE0117 before; `Microflows$Find` on `MyModule.OrderLine.Code`, mx check 0 errors, and describe → exec "Unchanged" after. - **A navigation profile's application title and icon can be set and are described** (ako/mxcli#1756, mendixlabs/mxcli#1664) — the title (Studio Pro's *Application title*, `Mendix` in a new project) is what the runtime puts in every browser tab as `Mendix - `, and it had no MDL form: `create or modify navigation Responsive app title 'EWO' …` failed with "extraneous input 'app'", `describe navigation` printed neither it nor the *Application icon*, and `create or modify translations … for en_US` refuses the source language, so every generated app kept "Mendix" in the tab. `create or modify navigation` now takes `app title ''` and `app icon ` (`app icon none` clears it), in any order with the other clauses; `app` and `icon` stay usable as names. The title is written as the project default language's item, the title's other languages kept; the icon is resolved like any image reference, by `check --references` and before exec writes; an omitted clause keeps the stored value; describe prints both. A native profile has neither, so both are refused there. - **A test whose earlier `@expect` fails is no longer reported PASS on the after-startup runner** (ako/mxcli#1753) — every Docker run and `--local --legacy-runner` logged a verdict line per assertion, and the result was taken from the last one: a failing first assertion followed by a passing second reported PASS, with the failure message erased, and `mxcli test` exited 0. A failure now sticks, and the test reports the first one, plus how many more failed. The endpoint runner (`--local`, `--attach`, `--watch`) was not affected. - **Two tests that assign the same variable with a lower-case `call` or `create` run on the after-startup runner** (found with ako/mxcli#1753) — that runner compiles every test into one microflow and renames each test's variables, but it recognised only an upper-case `CALL` or `CREATE`. `$r = call microflow …`, the spelling mdl 1 and `fmt` write, was left as it was, so two tests assigning `$r` declared it twice and injection failed with MDL063 (CE0111 from mxbuild). 8 of the 16 test files in `mdl-examples/` hit this under Docker or `--legacy-runner`. A variable first assigned by `retrieve`, `declare` or a list operation is still not renamed there. diff --git a/mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl b/mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl new file mode 100644 index 000000000..3c21e20a6 --- /dev/null +++ b/mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl @@ -0,0 +1,61 @@ +mdl 1; +-- ako/mxcli#1769 (mendixlabs/mxcli#1673) — a find in a loop an alter writes +-- +-- `alter microflow … { replace loop … with begin … end; }` whose loop body +-- finds by member in a list retrieved over an association before the loop +-- (`retrieve $Lines from $Order/MyModule.OrderLine_Order`) stored the find as +-- Microflows$FindByExpression with the bare text `Code = $Lookup/Code`: check +-- --references and exec reported nothing, and mx check failed CE0117 "Error(s) +-- in expression." `create microflow` stores the same find as Microflows$Find +-- on MyModule.OrderLine.Code. +-- +-- Two causes. The member guard a spliced fragment has (ako/mxcli#891) was not +-- inherited by the builder of a body nested in it — a loop, a while loop, an +-- error handler — so it wrote a member it could not qualify where the main +-- line refused it. And the fragment did not know what a stored retrieve over +-- an association holds, so it could not qualify the member. A retrieve from +-- the TO end of a reference (Order here) yields the list of the FROM entity +-- (OrderLine); from the FROM end, the one TO object. +-- +-- Run on any project: +-- mxcli exec mdl-examples/bug-tests/1769-alter-loop-find-over-association-list.mdl -p app.mpr +-- mxcli bson dump -p app.mpr --type microflow --object MyModule.SUB_MatchLines +-- Expected: "Altered microflow MyModule.SUB_MatchLines"; the dump shows +-- "$Type": "Microflows$Find" with "Attribute": "MyModule.OrderLine.Code" and +-- "Expression": "$Lookup/Code"; mx check reports no error for the flow, and +-- exec of `describe microflow MyModule.SUB_MatchLines` reports Unchanged. +-- Before the fix: Microflows$FindByExpression "Code = $Lookup/Code" and CE0117 +-- (measured on 11.13.0). + +create module MyModule; +create persistent entity MyModule.Order (Code: String(50)); +create persistent entity MyModule.OrderLine (Code: String(50), Status: String(20)); +create association MyModule.OrderLine_Order from MyModule.OrderLine to MyModule.Order; +create non-persistent entity MyModule.Lookup (Code: String(50)); + +create microflow MyModule.SUB_MatchLines ($Order: MyModule.Order, $Lookups: List of MyModule.Lookup) returns Integer as $Found +begin + declare $Found Integer = 0; + retrieve $Lines from $Order/MyModule.OrderLine_Order; + loop $Lookup in $Lookups + begin + $Line = find $Lines by Code = $Lookup/Code; + if $Line != empty then + set $Found = $Found + 1; + end if; + end loop; + return $Found; +end; + +-- The loop replaced with its own text. +alter microflow MyModule.SUB_MatchLines { + replace loop $Lookup in $Lookups with begin + loop $Lookup in $Lookups + begin + $Line = find $Lines by Code = $Lookup/Code; + if $Line != empty then + set $Found = $Found + 1; + end if; + end loop; + end; +}; diff --git a/mdl/executor/cmd_alter_flow.go b/mdl/executor/cmd_alter_flow.go index 30d6c192e..1205acecf 100644 --- a/mdl/executor/cmd_alter_flow.go +++ b/mdl/executor/cmd_alter_flow.go @@ -987,7 +987,18 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared for _, p := range a.mf.Parameters { add(p.Name, p.Type) } + // What is typed from another variable — a retrieve over an association + // from its start, a loop's iterator from its list — is typed once the + // rest are, below. + deps := storedDependents{} + fb := &flowBuilder{backend: ctx.Backend} for _, c := range a.cands { + if loop, ok := c.Object.(*microflows.LoopedActivity); ok { + if src, ok := loop.LoopSource.(*microflows.IterableList); ok { + deps.add(src.VariableName, storedDependent{from: src.ListVariableName}) + } + continue + } act, ok := c.Object.(*microflows.ActionActivity) if !ok || c.OutputVariable == "" { continue @@ -1005,7 +1016,13 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared } case *microflows.RetrieveAction: // A database retrieve names its entity; one over an association - // depends on which side it starts from and stays untyped here. + // yields an end of it, which depends on the end it starts from + // and is typed below (ako/mxcli#1769). + if src, ok := x.Source.(*microflows.AssociationRetrieveSource); ok { + declared[c.OutputVariable] = "Unknown" + deps.add(c.OutputVariable, storedDependent{from: src.StartVariable, assoc: src.AssociationQualifiedName}) + break + } src, ok := x.Source.(*microflows.DatabaseRetrieveSource) entity := "" if ok { @@ -1026,7 +1043,7 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared // A call's result has the called microflow's return type. var rt microflows.DataType if x.MicroflowCall != nil { - rt = (&flowBuilder{backend: ctx.Backend}).lookupMicroflowReturnType(x.MicroflowCall.Microflow) + rt = fb.lookupMicroflowReturnType(x.MicroflowCall.Microflow) } switch rt.(type) { case *microflows.ObjectType, *microflows.ListType: @@ -1038,9 +1055,72 @@ func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared declared[c.OutputVariable] = "Unknown" } } + deps.typeAll(fb, varTypes, declared) return varTypes, declared } +// storedDependent is a stored variable whose type follows from another's: the +// output of a retrieve over assoc from the variable from, or — assoc empty — +// the iterator of a loop over the list from. +type storedDependent struct { + from, assoc string +} + +// storedDependents are the stored flow's dependent variables by name. A name +// two of them give different origins is nil: which one a fragment means +// depends on where it goes (two loops can share an iterator name), and a guess +// would qualify a member with the wrong entity. It stays untyped and the +// member is refused. +type storedDependents map[string]*storedDependent + +func (d storedDependents) add(name string, dep storedDependent) { + if name == "" { + return + } + if old, seen := d[name]; seen { + if old == nil || *old != dep { + d[name] = nil + } + return + } + d[name] = &dep +} + +// typeAll types every dependent whose origin is typed — a parameter, a +// database retrieve, another dependent — whatever order the stored flow lists +// them in, until a pass types nothing new. A name the rest of the stored flow +// already types keeps that type. +func (d storedDependents) typeAll(fb *flowBuilder, varTypes, declared map[string]string) { + for progress := true; progress; { + progress = false + for name, dep := range d { + if dep == nil { + continue + } + if _, typed := varTypes[name]; typed { + continue + } + from, ok := varTypes[dep.from] + if !ok { + continue + } + t := "" + if dep.assoc == "" { + if elem, isList := strings.CutPrefix(from, "List of "); isList { + t = elem + } + } else { + t = fb.associationRetrieveType(from, dep.assoc) + } + if t != "" { + varTypes[name] = t + delete(declared, name) + progress = true + } + } + } +} + // systemVariables are in scope everywhere they exist at all; the platform // reports a misuse (a $latestError outside an error handler) itself. var systemVariables = map[string]bool{ diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index cbdf6f816..55dd0c3c8 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -2833,6 +2833,51 @@ func (fb *flowBuilder) lookupAssociation(moduleName, assocName string) *assocLoo return nil } +// associationRetrieveType is what `retrieve $X from $Start/assocQN` binds when +// $Start holds startType ("Module.Entity"): the list or the object at the +// other end, by associationRetrieveYieldsList — the answer addRetrieveAction +// and check --references take. Mind the inversion: an association's parent is +// its FROM end and its child the TO end, so a retrieve from the TO end yields +// the FROM entity. "" when it cannot be established: the association or the +// start unknown, the start a list, or the start no end of it — or, over a +// reference, both ends (a self-reference), where which end it starts from is +// not known. +func (fb *flowBuilder) associationRetrieveType(startType, assocQN string) string { + if startType == "" || strings.HasPrefix(startType, "List of ") { + return "" + } + module, name, ok := strings.Cut(assocQN, ".") + if !ok || module == "" || name == "" { + return "" + } + info := fb.lookupAssociation(module, name) + if info == nil || info.parentEntityQN == "" || info.childEntityQN == "" { + return "" + } + refSet := info.Type == domainmodel.AssociationTypeReferenceSet + fromTo := fb.entityIsSubtypeOf(startType, info.childEntityQN) + fromFrom := fb.entityIsSubtypeOf(startType, info.parentEntityQN) + switch { + case fromTo && fromFrom: + // Either end: a reference set to itself is a list of it whichever + // end it starts from, and nothing else is known. + if refSet && info.parentEntityQN == info.childEntityQN { + return "List of " + info.parentEntityQN + } + return "" + case !fromTo && !fromFrom: + return "" + } + other := info.childEntityQN + if fromTo { + other = info.parentEntityQN + } + if associationRetrieveYieldsList(refSet, info.Owner == domainmodel.AssociationOwnerBoth, fromTo) { + return "List of " + other + } + return other +} + // qualifyIteratorAttributes rewrites a bare attribute name in a FILTER/FIND // predicate into `$currentObject/`. // diff --git a/mdl/executor/flow_splice_assoc_retrieve_test.go b/mdl/executor/flow_splice_assoc_retrieve_test.go new file mode 100644 index 000000000..be5dd7aad --- /dev/null +++ b/mdl/executor/flow_splice_assoc_retrieve_test.go @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mfmutator" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ako/mxcli#1769 (mendixlabs/mxcli#1673): a fragment spliced into a stored +// flow is built knowing the variables the stored flow declares +// (storedVariables), and a retrieve over an association was left untyped +// there, though a database retrieve and a list parameter were typed. A find by +// member in a list retrieved over an association could not be qualified, so +// the fragment refused it — or, in a loop the fragment brought, wrote it as a +// find by the invalid expression `Code = …` (CE0117). + +// orderFixture is the reported domain model: OrderLine_Order runs FROM +// OrderLine TO Order (ParentPointer = OrderLine, ChildPointer = Order), so a +// retrieve over it from an Order yields the list of its lines and one from an +// OrderLine yields its one order. Order_Tag is a reference set FROM Order TO +// Tag; Order_Invoice a one-to-one (owner Both) reference FROM Order TO +// Invoice; Order_Parent a reference from Order to itself. +func orderFixture(t *testing.T) (*ExecContext, *alterFlowContext) { + t.Helper() + mod := mkModule("Mod") + attr := func(name string) *domainmodel.Attribute { + return &domainmodel.Attribute{BaseElement: model.BaseElement{ID: nextID("attr")}, Name: name} + } + order := mkEntity(mod.ID, "Order") + order.Attributes = []*domainmodel.Attribute{attr("Code")} + line := mkEntity(mod.ID, "OrderLine") + line.Attributes = []*domainmodel.Attribute{attr("Code"), attr("Status")} + lookup := mkEntity(mod.ID, "Lookup") + lookup.Attributes = []*domainmodel.Attribute{attr("Code")} + tag := mkEntity(mod.ID, "Tag") + tag.Attributes = []*domainmodel.Attribute{attr("Label")} + invoice := mkEntity(mod.ID, "Invoice") + invoice.Attributes = []*domainmodel.Attribute{attr("Number")} + dm := mkDomainModel(mod.ID, order, line, lookup, tag, invoice) + tags := mkAssociation(dm.ID, "Order_Tag", order.ID, tag.ID) + tags.Type = domainmodel.AssociationTypeReferenceSet + oneToOne := mkAssociation(dm.ID, "Order_Invoice", order.ID, invoice.ID) + oneToOne.Owner = domainmodel.AssociationOwnerBoth + dm.Associations = []*domainmodel.Association{ + mkAssociation(dm.ID, "OrderLine_Order", line.ID, order.ID), + tags, + oneToOne, + mkAssociation(dm.ID, "Order_Parent", order.ID, order.ID), + } + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetModuleByNameFunc: func(name string) (*model.Module, error) { + if name == "Mod" { + return mod, nil + } + return nil, nil + }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + + obj := func(e string) microflows.DataType { return µflows.ObjectType{EntityQualifiedName: "Mod." + e} } + over := func(out, start, assoc string) mfmutator.Candidate { + return mfmutator.Candidate{OutputVariable: out, Object: µflows.ActionActivity{Action: µflows.RetrieveAction{ + Source: µflows.AssociationRetrieveSource{StartVariable: start, AssociationQualifiedName: "Mod." + assoc}}}} + } + loop := func(iterator, list string) mfmutator.Candidate { + return mfmutator.Candidate{Object: µflows.LoopedActivity{ + LoopSource: µflows.IterableList{VariableName: iterator, ListVariableName: list}}} + } + a := &alterFlowContext{ + stmt: &ast.AlterFlowStmt{}, + mf: µflows.Microflow{Parameters: []*microflows.MicroflowParameter{ + {Name: "Order", Type: obj("Order")}, + {Name: "Line", Type: obj("OrderLine")}, + {Name: "Tag", Type: obj("Tag")}, + {Name: "Invoice", Type: obj("Invoice")}, + {Name: "Lookups", Type: µflows.ListType{EntityQualifiedName: "Mod.Lookup"}}, + }}, + entityNames: map[model.ID]string{order.ID: "Mod.Order", line.ID: "Mod.OrderLine"}, + cands: []mfmutator.Candidate{ + over("Lines", "Order", "OrderLine_Order"), // from the TO end: the lines + over("TheOrder", "Line", "OrderLine_Order"), // from the FROM end: the order + over("Tags", "Order", "Order_Tag"), // reference set, forward + over("TagOrders", "Tag", "Order_Tag"), // reference set, reverse + over("TheInvoice", "Order", "Order_Invoice"), // one-to-one, forward + over("InvoiceOrder", "Invoice", "Order_Invoice"), // one-to-one, reverse + // Starts from a variable a later candidate types: the order of the + // candidates is not the order a variable is typed in. + over("SiblingLines", "LineOrder", "OrderLine_Order"), + over("LineOrder", "Line", "OrderLine_Order"), + over("Parent", "Order", "Order_Parent"), // self-reference: which end is unknown + over("Nowhere", "Order", "Order_Missing"), // no such association + over("Unstarted", "NoSuchVariable", "OrderLine_Order"), + // A loop's iterator is an element of its list, and a retrieve in + // the body can start from it. + loop("EachLine", "Lines"), + over("EachOrder", "EachLine", "OrderLine_Order"), + // Two loops whose iterators share a name over different lists: + // which one a fragment means depends on where it goes. + loop("It", "Lines"), + loop("It", "Tags"), + over("ItOrder", "It", "OrderLine_Order"), + }, + } + return ctx, a +} + +// TestStoredVariables_AssociationRetrieveIsTyped: the output of a stored +// retrieve over an association has the type the association gives it from +// the end it starts at — a list of the FROM entity from the TO end, the one +// TO object from the FROM end, a list either way over a reference set, one +// object either way over a one-to-one. What cannot be established stays +// untyped, so a member of it is refused, not guessed. +func TestStoredVariables_AssociationRetrieveIsTyped(t *testing.T) { + ctx, a := orderFixture(t) + varTypes, declared := a.storedVariables(ctx) + want := map[string]string{ + "Lines": "List of Mod.OrderLine", + "TheOrder": "Mod.Order", + "Tags": "List of Mod.Tag", + "TagOrders": "List of Mod.Order", + "TheInvoice": "Mod.Invoice", + "InvoiceOrder": "Mod.Order", + "SiblingLines": "List of Mod.OrderLine", + "LineOrder": "Mod.Order", + "EachLine": "Mod.OrderLine", + "EachOrder": "Mod.Order", + } + for v, w := range want { + if got := varTypes[v]; got != w { + t.Errorf("$%s: typed %q, want %q", v, got, w) + } + } + for _, v := range []string{"Parent", "Nowhere", "Unstarted", "ItOrder"} { + if got, ok := varTypes[v]; ok { + t.Errorf("$%s: typed %q, want it left untyped", v, got) + } + if declared[v] != "Unknown" { + t.Errorf("$%s: declared %q, want Unknown", v, declared[v]) + } + } + if got, ok := varTypes["It"]; ok { + t.Errorf("$It, the iterator of two loops over different lists: typed %q, want it left untyped", got) + } +} + +// TestSpliceFragment_ReplacedLoopFindsOverAssociationList is the reported +// alter: a replaced loop whose body finds in a list retrieved over an +// association before the loop is written as a find by attribute, the way +// `create microflow` writes it — not as a find by the expression `Code = …`. +func TestSpliceFragment_ReplacedLoopFindsOverAssociationList(t *testing.T) { + ctx, a := orderFixture(t) + frag, err := a.buildFragment(ctx, parsedFlowBody(t, `mdl 1; +create microflow Mod.Fragment () +begin + loop $Lookup in $Lookups + begin + $Line = find $Lines by Code = $Lookup/Code; + change $TheOrder (Code = $Lookup/Code); + end loop; +end;`)) + if err != nil { + t.Fatalf("the replaced loop: %v", err) + } + ops := fragmentListOperations(frag.Objects) + if len(ops) != 1 { + t.Fatalf("want one list operation, got %d", len(ops)) + } + f, ok := ops[0].(*microflows.FindByAttributeOperation) + if !ok || f.Attribute != "Mod.OrderLine.Code" || f.Expression != "$Lookup/Code" { + t.Fatalf("the find was written as %#v, want a find by attribute Mod.OrderLine.Code = $Lookup/Code", ops[0]) + } + var changed []string + for _, obj := range frag.Objects { + loop, ok := obj.(*microflows.LoopedActivity) + if !ok { + continue + } + for _, inner := range loop.ObjectCollection.Objects { + if act, ok := inner.(*microflows.ActionActivity); ok { + if ch, ok := act.Action.(*microflows.ChangeObjectAction); ok { + for _, m := range ch.Changes { + changed = append(changed, m.AttributeQualifiedName) + } + } + } + } + } + if strings.Join(changed, ",") != "Mod.Order.Code" { + t.Errorf("the change over the forward retrieve wrote %v, want [Mod.Order.Code]", changed) + } +} diff --git a/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go index 4cc62bd1f..11680b697 100644 --- a/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go +++ b/mdl/executor/flow_splice_nested_guard_1769_pedapp_test.go @@ -4,10 +4,12 @@ package executor import ( "bytes" + "context" "strings" "testing" "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" ) // ako/mxcli#1769 (mendixlabs/mxcli#1673) on the Studio Pro-authored PedApp, @@ -122,3 +124,111 @@ func TestAlterMicroflow_FindOverUntypedListIsRefused(t *testing.T) { }) } } + +// matchLinesFlow returns the stored MyModule.SUB_MatchLines. +func matchLinesFlow(t *testing.T, exec *Executor) *microflows.Microflow { + t.Helper() + ctx := exec.newExecContext(context.Background()) + h, err := getHierarchy(ctx) + if err != nil { + t.Fatal(err) + } + all, err := ctx.Backend.ListMicroflows() + if err != nil { + t.Fatal(err) + } + for _, m := range all { + if m.Name == "SUB_MatchLines" && h.GetModuleName(h.FindModuleID(m.ContainerID)) == "MyModule" { + mf, err := ctx.Backend.GetMicroflow(m.ID) + if err != nil { + t.Fatal(err) + } + return mf + } + } + t.Fatal("MyModule.SUB_MatchLines not found") + return nil +} + +// TestAlterMicroflow_FindOverAssociationListIsByAttribute: every alter that +// writes `find $Lines by Code = …`, where $Lines is retrieved over an +// association before it, writes Microflows$Find on MyModule.OrderLine.Code, +// as `create microflow` does; check predicts it and exec writes it. +func TestAlterMicroflow_FindOverAssociationListIsByAttribute(t *testing.T) { + for name, src := range matchLinesFindIn("$Lines") { + t.Run(name, func(t *testing.T) { + exec, _, _ := openMatchLinesFixture(t) + predictedAgree(t, exec, src, "") + var finds int + for _, op := range fragmentListOperations(matchLinesFlow(t, exec).ObjectCollection.Objects) { + switch f := op.(type) { + case *microflows.FindOperation: + t.Errorf("a find over $%s was stored by the expression %q", f.ListVariable, f.Expression) + case *microflows.FindByAttributeOperation: + finds++ + if f.Attribute != "MyModule.OrderLine.Code" { + t.Errorf("a find over $%s is on %q, want MyModule.OrderLine.Code", f.ListVariable, f.Attribute) + } + } + } + if finds == 0 { + t.Errorf("no find by attribute was stored") + } + }) + } +} + +// The reported alter writes what describe prints back unchanged: exec of the +// describe output reports Unchanged. +func TestAlterMicroflow_ReplacedLoopDescribeRoundTrips(t *testing.T) { + exec, out, _ := openMatchLinesFixture(t) + predictedAgree(t, exec, matchLinesFindIn("$Lines")["replace loop"], "") + var desc bytes.Buffer + saved := exec.output + exec.output = &desc + err := afRun(t, exec, "describe microflow MyModule.SUB_MatchLines;") + exec.output = saved + if err != nil { + t.Fatal(err) + } + if !strings.Contains(desc.String(), "find $Lines by Code = $Lookup/Code;") { + t.Errorf("describe does not print the find by attribute:\n%s", desc.String()) + } + out.Reset() + if err := afRun(t, exec, "mdl 1;\n"+desc.String()); err != nil { + t.Fatalf("exec of the describe output: %v\n%s", err, out.String()) + } + if !strings.Contains(out.String(), "Unchanged") { + t.Errorf("exec of the describe output must report Unchanged, got:\n%s", out.String()) + } +} + +// A stored loop's iterator is typed from its list, so a member of it in a +// fragment inserted in the loop body is qualified; it was refused. +func TestAlterMicroflow_ChangeOnStoredIteratorIsQualified(t *testing.T) { + exec, _, _ := openMatchLinesFixture(t) + predictedAgree(t, exec, `mdl 1; +alter microflow MyModule.SUB_MatchLines { + insert after $Line begin change $Lookup (Code = 'x'); end; +};`, "") + var changed []string + var walk func(objs []microflows.MicroflowObject) + walk = func(objs []microflows.MicroflowObject) { + for _, obj := range objs { + switch o := obj.(type) { + case *microflows.ActionActivity: + if ch, ok := o.Action.(*microflows.ChangeObjectAction); ok { + for _, m := range ch.Changes { + changed = append(changed, m.AttributeQualifiedName) + } + } + case *microflows.LoopedActivity: + walk(o.ObjectCollection.Objects) + } + } + } + walk(matchLinesFlow(t, exec).ObjectCollection.Objects) + if strings.Join(changed, ",") != "MyModule.Lookup.Code" { + t.Errorf("the change was stored on %v, want [MyModule.Lookup.Code]", changed) + } +} From 47ec0e02fac3dfe0737ce3a2c6e18d91adc6b85d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:04:49 +0000 Subject: [PATCH 35/44] fix(canon): store a user task's targeting as UserSource below Mendix 11.2 Symptom (ako/mxcli#1771): on Mendix 10.12 to 11.1 every user task mxcli wrote failed mx check with CE1859 "XPath constraint is required". That included tasks targeted by a microflow, and multi-user tasks. exec and mxcli check were clean. `mx convert -p` 10.24.24 then stored an empty XPathBasedUserSource in place of each targeting, so the targeting was silently lost. Cause: userTaskToGen writes UserTargeting, which Mendix introduced in 11.2.0 (mendixmodelsdk: UserTaskActivity.userSource deleted and userTargeting introduced in 11.2.0). It is declared on the abstract UserTaskActivity, and the property floor looks up the concrete type only, so the key went into every version. 10.24 ignores it and fills the required UserSource with its default. Older shape, measured: two Studio Pro-authored Marketplace samples, Employee Onboarding (9.1.1, XPath) and Multilevel Dynamic Workflow (9.24.3, microflow), converted 9.24.30 -> 10.24.24 -> 11.13.0. The targeting keeps its $ID and value byte for byte. Only its $Type and the key holding it change: XPathBasedUserSource <-> XPathUserTargeting, MicroflowBasedUserSource <-> MicroflowUserTargeting, and EmptyUserSource -> NoUserTargeting (this fix's 10.24 output converted up). Fix: canon.LowerToProjectShape renames the targeting to the user source below 11.2, in its place, for every writer. Group targeting is 11.2 and has no older form, so it is refused, naming the task. Measured on fresh apps, exec of the bug-test then mx check: - 10.24.24.119653: before CE1859 x3, after 0 errors. mx convert 10.24.24 changes no key and no value. describe -> exec "Unchanged workflow". - 11.13.0: 0 errors, and the key sets match the unfixed build. Control: stubbing the lowering fails TestNormalize_UserTaskTargetingBelowMendix11_2 24 times, with UserTargeting still on the task. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...-10-29-user-task-targeting-below-11-2.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../1771-user-task-targeting-below-11-2.mdl | 69 +++++ .../modelsdk/workflow_usertask_shape_test.go | 90 ++++++ modelsdk/canon/versionshape.go | 11 +- modelsdk/canon/versionshape_usertask.go | 93 +++++++ .../mpr/writer_version_shape_1771_test.go | 256 ++++++++++++++++++ 8 files changed, 522 insertions(+), 1 deletion(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-targeting-below-11-2.json create mode 100644 mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl create mode 100644 mdl/backend/modelsdk/workflow_usertask_shape_test.go create mode 100644 modelsdk/canon/versionshape_usertask.go create mode 100644 modelsdk/mpr/writer_version_shape_1771_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-targeting-below-11-2.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-targeting-below-11-2.json new file mode 100644 index 000000000..fd5df2999 --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-targeting-below-11-2.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-29", "symptom": "On Mendix 10.12 to 11.1 (measured on 10.12.19 and 10.24.24) every user task mxcli wrote failed mx check with CE1859 \"XPath constraint is required\". That included tasks targeted by a microflow, and multi-user tasks. `mxcli check` and `exec` were clean, and `mx convert -p` 10.24.24 rewrote each task with an empty XPathBasedUserSource, so the targeting was silently lost", "cause": "userTaskToGen writes the targeting as UserTargeting (XPathUserTargeting / MicroflowUserTargeting / NoUserTargeting), the 11.2 property. Before 11.2 a user task stores UserSource (XPathBasedUserSource / MicroflowBasedUserSource / EmptyUserSource) (mendixmodelsdk: UserTaskActivity.userSource deleted, userTargeting introduced 11.2.0). Both are declared on the abstract Workflows$UserTaskActivity, and the floor looks up the concrete type only, so it never saw them. 10.24 loads the task and ignores the unknown key, then fills the required UserSource with its default, an XPath source with no constraint", "file": "`modelsdk/canon/versionshape_usertask.go` (lowerUserTargeting, userSourceOf), hooked in `modelsdk/canon/versionshape.go` (LowerToProjectShape: userSource below 11.2)", "insight": "**Read a version's ERROR as a read of the default, not of what was written.** CE1859 named the XPath constraint even on microflow-targeted tasks. 10.24 had dropped the whole key and read the required property's default, an XPath source. So 'XPath constraint is required' on a task with no XPath points at the property holding it, not the constraint. **The convert chain is the oracle for a renamed property**: Marketplace samples (Employee Onboarding 117785, 9.1.1 XPath; Multilevel Dynamic Workflow 221741, 9.24.3 microflow) converted 9.1.1 → 9.24.30 → 10.24.24 → 11.13.0 keep the targeting's $ID and value byte for byte and change only $Type and the holding key. So the lowering is a rename. 9.1.1 does not convert straight to 10.24 ('version not supported'), so go through 9.24.30 first. Converting this fix's 10.24 output up to 11.13 turns EmptyUserSource into NoUserTargeting, which confirms the third pair. Workflow Commons (2.6.0, 3.7.0, 3.8.0) holds no workflow at all. Group targeting has no older form and is refused naming the task. Measured: bug-test on a fresh 10.24.24 app, before CE1859 x3 and after 0 errors. `mx convert -p` 10.24.24 changes no key and no value. describe → exec prints 'Unchanged workflow'. 11.13.0 gives 0 errors with the same key sets before and after. Control: stubbing the lowering fails TestNormalize_UserTaskTargetingBelowMendix11_2 24 times with UserTargeting still on the task. Tests: TestNormalize_UserTaskTargetingBelowMendix11_2, TestNormalize_RefusesGroupTargetingBelowMendix11_2, TestWorkflowRead_UserTaskTargetingInTheMendix10Shape; mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl", "refs": ["#1771"], "ce": ["CE1859"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 6390f9622..eab978aac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. - **The error handlers of two consecutive activities no longer share a row** (ako/mxcli#1768, mendixlabs/mxcli#1672) — without `@position` a custom error handler is a row 130 px under its activity, running right, and the main line carries on one spacing after the activity however long that row is. When the next activity had a handler too, its row started at its own x, inside the first: two REST calls with three-element handlers stored the first handler's end event and the second handler's first activity both at 680;330. Neither check saw it — `layout flows --dry-run` re-derives with the same builder and called the flow already laid out, and MPR008 did not compare end events. A handler's row now occupies the lane under the main line the way a guard's branch already did, so whatever reaches down into that lane — the next activity with a handler, a decision, a loop — waits for it to clear, and a plain activity does not. The reported flow keeps `$First`, its row and `$Body` where they were and places `$Second` at 790;200, its row at 790..1110;330. The measurer counts the row too, so a loop or a branch holding handler rows is sized to them: a loop whose last handler row reached past its last activity was built 110 px wider than it was measured. Measured on 182 flows built from the `02*` and `06` doctype examples: 2 move — each an IF that followed a handler row lying under its split — and none gains an overlap; `mx check` 0 errors on 11.13.0 before and after. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 8350ab77b..0af20a6fa 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -187,6 +187,7 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | | The end of a parallel split path (`Workflows$EndOfParallelSplitPathActivity`, a type introduced in 10.21) | 10.21 | Not stored: a path holds its activities and nothing more (Studio Pro 9.4) | +| A user task's targeting (`targeting users xpath […]`, `targeting users microflow …`, none) | 11.2 | A `UserSource` (`XPathBasedUserSource`, `MicroflowBasedUserSource`, `EmptyUserSource`) holding the same value, instead of a `UserTargeting` ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)) | | A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | | A page's flow call argument that is the enclosing object (`call microflow M.F(P = $currentObject)`, `call nanoflow …`) | 10.10 | Not stored: Studio Pro infers it. A page parameter, snippet parameter or widget argument is stored as before, as a `Variable` | @@ -197,6 +198,7 @@ What an older shape cannot hold is refused, naming the element, rather than drop - a data view fed by another widget below 10.0; - a private constant value in a configuration below 10.9; - a page's flow call argument that is any other expression (a literal, an attribute path) below 10.10; +- a user task targeting groups (`targeting groups xpath …`, `targeting groups microflow …`) below 11.2, where a task targets users only ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl b/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl new file mode 100644 index 000000000..c23a49cc1 --- /dev/null +++ b/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl @@ -0,0 +1,69 @@ +mdl 1; +-- ako/mxcli#1771: on Mendix 10.12 to 11.1 every user task mxcli wrote failed +-- mx check with CE1859 "XPath constraint is required". That included tasks +-- targeted by a microflow. `exec` and `mxcli check` were clean. +-- +-- mxcli stored the targeting as UserTargeting, the 11.2 property. Before 11.2 a +-- user task stores it as UserSource. UserTargeting is declared on the abstract +-- UserTaskActivity, which the property floor does not look up, so the key went +-- into every version. The older versions read the task as having no user +-- source. `mx convert -p` 10.24.24 then dropped the targeting and stored an +-- empty XPathBasedUserSource in its place, so the targeting was silently lost. +-- +-- The older shape, measured with `mx convert -p`. The sources are Studio +-- Pro-authored Marketplace samples: Employee Onboarding (9.1.1, XPath +-- targeting) and Multilevel Dynamic Workflow (9.24.3, microflow targeting). +-- Each was converted 9.24.30 → 10.24.24 → 11.13.0. The value is carried byte +-- for byte, and only its holder changes: +-- UserSource: XPathBasedUserSource ↔ UserTargeting: XPathUserTargeting +-- UserSource: MicroflowBasedUserSource ↔ UserTargeting: MicroflowUserTargeting +-- A task with no targeting is an EmptyUserSource (9.10+), and from 11.2 a +-- NoUserTargeting. Group targeting is 11.2. Before 11.2 it is refused with +-- MDL-VERSION01 by check -p, and at write time. +-- +-- Measured on fresh apps (`mxcli new … --skip-build`): exec, then +-- `mxcli docker check`. +-- 10.24.24.119653 before: CE1859 on every task. After: 0 errors. +-- `mx convert -p` 10.24.24 changes no key and no value. +-- describe → exec prints "Unchanged workflow". +-- 11.13.0 0 errors before and after, with the same key sets. + +create module Probe1771Tg; + +create persistent entity Probe1771Tg.Request ( Title: String(100) ); + +create microflow Probe1771Tg.ACT_GetUsers ($Workflow: System.Workflow, $Context: Probe1771Tg.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +create page Probe1771Tg.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txt1 (content: 'Task') +}; + +create workflow Probe1771Tg.WF_Targeting + parameter $WorkflowContext: Probe1771Tg.Request + display 'Targeting' +begin + user task ByXPath 'By XPath' + page Probe1771Tg.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Done' { }; + user task ByMicroflow 'By microflow' + page Probe1771Tg.TaskPage + targeting users microflow Probe1771Tg.ACT_GetUsers + outcomes 'Done' { }; + user task Untargeted 'No targeting' + page Probe1771Tg.TaskPage + outcomes 'Done' { }; + multi user task MultiByXPath 'Multi by XPath' + page Probe1771Tg.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Approve' { } 'Reject' { }; +end workflow; diff --git a/mdl/backend/modelsdk/workflow_usertask_shape_test.go b/mdl/backend/modelsdk/workflow_usertask_shape_test.go new file mode 100644 index 000000000..34d2ce89f --- /dev/null +++ b/mdl/backend/modelsdk/workflow_usertask_shape_test.go @@ -0,0 +1,90 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "reflect" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/modelsdk/canon" + "github.com/mendixlabs/mxcli/modelsdk/codec" + genWf "github.com/mendixlabs/mxcli/modelsdk/gen/workflows" + "github.com/mendixlabs/mxcli/modelsdk/version" + "github.com/mendixlabs/mxcli/sdk/workflows" +) + +// ako/mxcli#1771: below 11.2 a user task's targeting is stored as UserSource, +// the shape canon.LowerToProjectShape writes on every unit write. describe +// reads what is stored, so a task written to such a project must read back as +// the targeting the statement gave it — or describe → exec drops it. + +// storedUserTask writes act as a workflow for a project of version pv — the +// encoder, then the lowering and the floor every unit write passes — and reads +// the stored document back. +func storedUserTask(t *testing.T, act *workflows.UserTask, pv *types.ProjectVersion) *workflows.UserTask { + t.Helper() + wf := &workflows.Workflow{ + Name: "WF", + Parameter: &workflows.WorkflowParameter{EntityRef: "M.Ctx"}, + Flow: &workflows.Flow{Activities: []workflows.WorkflowActivity{act}}, + } + raw, err := encodeWorkflow(wf, pv) + if err != nil { + t.Fatalf("encode: %v", err) + } + v := version.Version{Major: pv.MajorVersion, Minor: pv.MinorVersion, Patch: pv.PatchVersion} + if raw, err = canon.LowerToProjectShape(raw, &v); err != nil { + t.Fatalf("%s: lower: %v", v, err) + } + if raw, err = canon.StripUndeclaredProperties(raw, &v); err != nil { + t.Fatalf("%s: floor: %v", v, err) + } + el, err := codec.NewDecoder(codec.DefaultRegistry).Decode(raw) + if err != nil { + t.Fatalf("decode: %v", err) + } + f, _ := el.(*genWf.Workflow).Flow().(*genWf.Flow) + back := workflowFlowFromGen(f) + if len(back.Activities) != 1 { + t.Fatalf("%s: read %d activities, want 1", v, len(back.Activities)) + } + ut, ok := back.Activities[0].(*workflows.UserTask) + if !ok { + t.Fatalf("%s: read back %T, want a user task", v, back.Activities[0]) + } + return ut +} + +func TestWorkflowRead_UserTaskTargetingInTheMendix10Shape(t *testing.T) { + for _, tc := range []struct { + name string + source workflows.UserSource + }{ + {"users microflow", &workflows.MicroflowBasedUserSource{Microflow: "M.ACT_Users"}}, + {"users xpath", &workflows.XPathBasedUserSource{XPath: "[Name = 'admin']"}}, + {"none", &workflows.NoUserSource{}}, + } { + for _, pv := range []*types.ProjectVersion{ + {MajorVersion: 10, MinorVersion: 12, PatchVersion: 19}, + {MajorVersion: 10, MinorVersion: 24, PatchVersion: 24}, + {MajorVersion: 11, MinorVersion: 1}, + {MajorVersion: 11, MinorVersion: 13}, // control: the 11.2 shape + } { + for _, multi := range []bool{false, true} { + ut := &workflows.UserTask{Page: "M.TaskPage", IsMulti: multi, UserSource: tc.source} + ut.Name, ut.Caption = "Review", "Review" + ut.Outcomes = []*workflows.UserTaskOutcome{{Value: "Approve"}, {Value: "Reject"}} + + got := storedUserTask(t, ut, pv) + if got.IsMulti != multi { + t.Errorf("%s %d.%d multi=%v: read back multi=%v", tc.name, pv.MajorVersion, pv.MinorVersion, multi, got.IsMulti) + } + if !reflect.DeepEqual(got.UserSource, tc.source) { + t.Errorf("%s %d.%d multi=%v: targeting read back as %#v, want %#v", + tc.name, pv.MajorVersion, pv.MinorVersion, multi, got.UserSource, tc.source) + } + } + } + } +} diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 7ac90cb7b..4b51ae071 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -59,6 +59,9 @@ import ( // Marketplace's "Example project workflow extension" 2.0.0), and a project // holding the marker does not load (TypeCacheUnknownTypeException). The // floor checks property versions, never a type's own. +// - Workflows$ user task targeting: UserTargeting from 11.2.0, declared on +// the abstract UserTaskActivity; before, UserSource (ako/mxcli#1771). See +// lowerUserTargeting. // // It runs on every unit write, before the floor, so no writer needs to know // these shapes. What the older shape cannot hold — a design property value @@ -79,9 +82,10 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { flatConstant: !PropertyDeclared("Settings$ConstantValue", "SharedOrPrivateValue", *pv), variableArgs: !PropertyDeclared("Forms$MicroflowParameterMapping", "Expression", *pv), unmarkedPaths: pv.Compare(endOfParallelSplitPathSince) < 0, + userSource: !PropertyDeclared("Workflows$UserTaskActivity", "UserTargeting", *pv), } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths { + !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths && !l.userSource { return contents, nil } var doc bson.D @@ -103,6 +107,7 @@ type shapeLowering struct { flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool flatConstant, variableArgs, unmarkedPaths bool + userSource bool changed bool } @@ -145,6 +150,10 @@ func (l *shapeLowering) walk(v any) (any, error) { if x, err = l.lowerConstantValue(x); err != nil { return nil, err } + case l.userSource && userTaskTypes[typ]: + if x, err = l.lowerUserTargeting(x); err != nil { + return nil, err + } } if l.noFullObjects { if x, err = l.dropForceFullObjects(x, typ); err != nil { diff --git a/modelsdk/canon/versionshape_usertask.go b/modelsdk/canon/versionshape_usertask.go new file mode 100644 index 000000000..8ce65d694 --- /dev/null +++ b/modelsdk/canon/versionshape_usertask.go @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + "fmt" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/version" +) + +// The workflow user task's older shapes (ako/mxcli#1771), lowered here rather +// than in the workflow encoder because `alter workflow` inserts and edits user +// tasks through the raw mutator, which never passes the encoder: every unit +// write passes LowerToProjectShape. + +// userTaskTypes are the $Types a user task is stored as, in any version. +var userTaskTypes = map[string]bool{ + "Workflows$SingleUserTaskActivity": true, + "Workflows$MultiUserTaskActivity": true, + "Workflows$UserTask": true, +} + +// userSourceOf maps a user targeting (11.2+) to the user source an older +// version stores the same fact in. Measured with `mx convert -p` on +// Studio Pro-authored workflows (Employee Onboarding 9.1.1, Multilevel Dynamic +// Workflow 9.24.3) taken 9.24.30 → 10.24.24 → 11.13.0: the targeting keeps its +// $ID and its one value byte for byte, only its $Type and the key holding it +// change. Group targeting has no user source: it is 11.2, with the groups it +// targets. +var userSourceOf = map[string]string{ + "Workflows$XPathUserTargeting": "Workflows$XPathBasedUserSource", + "Workflows$MicroflowUserTargeting": "Workflows$MicroflowBasedUserSource", + "Workflows$NoUserTargeting": "Workflows$EmptyUserSource", +} + +// emptyUserSourceSince is the version that introduced Workflows$EmptyUserSource, +// a user task with no targeting (mendixmodelsdk 4.117.0: introduced "9.10.0"). +var emptyUserSourceSince = version.Version{Major: 9, Minor: 10} + +// lowerUserTargeting replaces a user task's UserTargeting with the UserSource a +// version before 11.2 stores, in its place. mxcli writes UserTargeting on every +// version, and the floor never saw it: it is declared on the abstract +// UserTaskActivity. 10.12.19 and 10.24.24 then read the task as having no user +// source and report CE1859 "XPath constraint is required" on it, whatever its +// targeting was, and `mx convert` stores an empty XPathBasedUserSource in its +// place — the targeting silently gone. Group targeting is refused, naming the +// task. +func (l *shapeLowering) lowerUserTargeting(task bson.D) (bson.D, error) { + i := shapeIndex(task, "UserTargeting") + if i < 0 { + return task, nil + } + targeting, _ := task[i].Value.(bson.D) + var source bson.D + if targeting != nil { + typ, _ := shapeString(targeting, "$Type") + name, _ := shapeString(task, "Name") + sourceType, ok := userSourceOf[typ] + switch { + case !ok: + return nil, fmt.Errorf("user task %q targets %s, which Mendix introduced in 11.2 (this project is %s): "+ + "a user task there targets users, by XPath or by a microflow", name, targetingKind(typ), l.pv) + case sourceType == "Workflows$EmptyUserSource" && l.pv.Compare(emptyUserSourceSince) < 0: + return nil, fmt.Errorf("user task %q has no targeting, which Mendix introduced in 9.10 (this project is %s): "+ + "give it `targeting users xpath […]` or `targeting users microflow …`", name, l.pv) + } + source = append(bson.D{}, targeting...) + source[shapeIndex(source, "$Type")].Value = sourceType + } + task[i] = bson.E{Key: "UserSource", Value: source} + l.changed = true + // A stored UserSource beside it (an older document an edit gave a + // UserTargeting) is the fact the edit replaced. + for j := range task { + if j != i && task[j].Key == "UserSource" { + return append(append(bson.D{}, task[:j]...), task[j+1:]...), nil + } + } + return task, nil +} + +// targetingKind names a targeting $Type as the statement that writes it does. +func targetingKind(typ string) string { + switch typ { + case "Workflows$XPathGroupTargeting": + return "groups by XPath (`targeting groups xpath`)" + case "Workflows$MicroflowGroupTargeting": + return "groups by a microflow (`targeting groups microflow`)" + } + return "with a " + typ +} diff --git a/modelsdk/mpr/writer_version_shape_1771_test.go b/modelsdk/mpr/writer_version_shape_1771_test.go new file mode 100644 index 000000000..1d1ccd49e --- /dev/null +++ b/modelsdk/mpr/writer_version_shape_1771_test.go @@ -0,0 +1,256 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "fmt" + "sort" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/mpr/version" +) + +// ako/mxcli#1771: a workflow user task as mxcli builds it is the 11.2 shape, +// and Mendix stored a user task differently before two version boundaries. +// Measured with `mx convert -p` on Studio Pro-authored projects (the +// Marketplace's Employee Onboarding, 9.1.1, XPath targeting; Multilevel Dynamic +// Workflow, 9.24.3, microflow targeting), converted 9.24.30 → 10.24.24 → 11.13.0: +// +// 9.x–10.11 Workflows$UserTask UserSource + UserTaskCompletion +// 10.12–11.1 Workflows$SingleUserTaskActivity UserSource +// 11.2+ Workflows$SingleUserTaskActivity UserTargeting +// +// The targeting value is carried byte-identical across both conversions; only +// its holder changes: XPathBasedUserSource ↔ XPathUserTargeting and +// MicroflowBasedUserSource ↔ MicroflowUserTargeting, each {$ID, $Type, value}. + +var eidSeq byte + +// eid is a distinct, non-zero element $ID, so a test can tell elements apart +// and follow a pointer. +func eid() bson.Binary { + eidSeq++ + return bson.Binary{Subtype: 0, Data: []byte{0x17, 0x71, eidSeq, 3, 4, 5, 0x46, 7, 0x88, 9, 10, 11, 12, 13, 14, eidSeq}} +} + +func xpathUserTargeting(xpath string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$XPathUserTargeting"}, {Key: "XPathConstraint", Value: xpath}} +} + +func microflowUserTargeting(mf string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$MicroflowUserTargeting"}, {Key: "Microflow", Value: mf}} +} + +func noUserTargeting() bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$NoUserTargeting"}} +} + +func stringTemplate(text string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Microflows$StringTemplate"}, + {Key: "Parameters", Value: bson.A{int32(2)}}, {Key: "Text", Value: text}} +} + +func userTaskOutcome(value string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$UserTaskOutcome"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: bson.A{int32(3)}}}}, + {Key: "PersistentId", Value: pid()}, {Key: "Value", Value: value}} +} + +// userTaskDoc is a user task as userTaskToGen builds it (below 11.1, where the +// encoder leaves out RelativeMiddlePoint and Size): the keys it writes, in its +// order, with the codec's BoundaryEvents and Annotation defaults last. multi +// adds AwaitAllUsers, a consensus CompletionCriteria falling back to the first +// outcome, and TargetUserInput, as a `multi user task` with no rule. +func userTaskDoc(name string, multi bool, targeting bson.D) bson.D { + typ := "Workflows$SingleUserTaskActivity" + if multi { + typ = "Workflows$MultiUserTaskActivity" + } + approve, reject := userTaskOutcome("Approve"), userTaskOutcome("Reject") + d := bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: typ}, + {Key: "AutoAssignSingleTargetUser", Value: false}} + if multi { + d = append(d, bson.E{Key: "AwaitAllUsers", Value: false}) + } + d = append(d, bson.E{Key: "Caption", Value: name}) + if multi { + approveID, _ := get(approve, "$ID") + d = append(d, bson.E{Key: "CompletionCriteria", Value: bson.D{{Key: "$ID", Value: eid()}, + {Key: "$Type", Value: "Workflows$ConsensusCompletionCriteria"}, {Key: "FallbackOutcomePointer", Value: approveID}}}) + } + d = append(d, + bson.E{Key: "DueDate", Value: ""}, + bson.E{Key: "Name", Value: name}, + bson.E{Key: "OnCreatedEvent", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$NoEvent"}}}, + bson.E{Key: "Outcomes", Value: bson.A{int32(3), approve, reject}}, + bson.E{Key: "PersistentId", Value: pid()}, + bson.E{Key: "TaskDescription", Value: stringTemplate("")}, + bson.E{Key: "TaskName", Value: stringTemplate(name)}, + bson.E{Key: "TaskPage", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$PageReference"}, {Key: "Page", Value: "M.TaskPage"}}}) + if multi { + d = append(d, bson.E{Key: "TargetUserInput", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$AllUserInput"}}}) + } + return append(d, + bson.E{Key: "UserTargeting", Value: targeting}, + bson.E{Key: "BoundaryEvents", Value: bson.A{int32(2)}}, + bson.E{Key: "Annotation", Value: nil}) +} + +// workflowWithUserTasks wraps tasks in a workflow unit. +func workflowWithUserTasks(tasks ...bson.D) bson.D { + acts := bson.A{int32(3)} + for _, t := range tasks { + acts = append(acts, t) + } + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Workflow"}, + {Key: "Flow", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$Flow"}, {Key: "Activities", Value: acts}}}, + {Key: "Name", Value: "WF"}, + {Key: "Title", Value: "WF"}} +} + +// findNamed returns the first document in v named name. +func findNamed(v any, name string) bson.D { + switch x := v.(type) { + case bson.D: + if n, _ := get(x, "Name"); n == name { + return x + } + for _, e := range x { + if d := findNamed(e.Value, name); d != nil { + return d + } + } + case bson.A: + for _, el := range x { + if d := findNamed(el, name); d != nil { + return d + } + } + } + return nil +} + +// sameID reports whether two stored $IDs (bson.Binary, not comparable with ==) +// are the same. +func sameID(a, b any) bool { + x, ok1 := a.(bson.Binary) + y, ok2 := b.(bson.Binary) + return ok1 && ok2 && bytes.Equal(x.Data, y.Data) +} + +func vlabel(v *version.ProjectVersion) string { + return fmt.Sprintf("%d.%d.%d", v.MajorVersion, v.MinorVersion, v.PatchVersion) +} + +func sortedKeys(d bson.D) string { + ks := keysOf(d) + sort.Strings(ks) + return strings.Join(ks, ",") +} + +// User targeting is UserTargeting from 11.2 and UserSource before it +// (mendixmodelsdk: UserTaskActivity.userSource deleted, userTargeting +// introduced 11.2.0). UserTargeting is declared on the abstract +// UserTaskActivity, which the floor does not look up, so it was written to +// every version: on 10.12.19 and 10.24.24 the task then had no UserSource, and +// mx check reported CE1859 "XPath constraint is required" on every user task — +// the microflow-targeted ones too — while `mx convert` 10.24.24 dropped the +// targeting and stored an empty XPathBasedUserSource in its place. +func TestNormalize_UserTaskTargetingBelowMendix11_2(t *testing.T) { + kinds := []struct { + name string + targeting func() bson.D + sourceType string + valueKey string + value string + }{ + {"users xpath", func() bson.D { return xpathUserTargeting("[Name = 'admin']") }, "Workflows$XPathBasedUserSource", "XPathConstraint", "[Name = 'admin']"}, + {"users microflow", func() bson.D { return microflowUserTargeting("M.ACT_GetUsers") }, "Workflows$MicroflowBasedUserSource", "Microflow", "M.ACT_GetUsers"}, + {"none", noUserTargeting, "Workflows$EmptyUserSource", "", ""}, + } + for _, tt := range []struct { + v *version.ProjectVersion + lowered bool + }{ + {pv(9, 24, 30), true}, + {pv(10, 12, 19), true}, + {pv(10, 24, 24), true}, + {pv(11, 1, 0), true}, + {pv(11, 2, 0), false}, + {pv(11, 13, 0), false}, // control + } { + for _, k := range kinds { + for _, multi := range []bool{false, true} { + targeting := k.targeting() + wantID, _ := get(targeting, "$ID") + d, err := normalizeShape(t, tt.v, workflowWithUserTasks(userTaskDoc("Task", multi, targeting))) + label := vlabel(tt.v) + " " + k.name + if multi { + label += " (multi)" + } + if err != nil { + t.Errorf("%s: refused: %v", label, err) + continue + } + task := findNamed(d, "Task") + _, hasTargeting := get(task, "UserTargeting") + srcAny, hasSource := get(task, "UserSource") + if !tt.lowered { + if !hasTargeting || hasSource { + t.Errorf("%s: keys %v, want UserTargeting and no UserSource", label, keysOf(task)) + } + continue + } + if hasTargeting || !hasSource { + t.Errorf("%s: keys %v, want UserSource and no UserTargeting", label, keysOf(task)) + continue + } + src, _ := srcAny.(bson.D) + if typ, _ := get(src, "$Type"); typ != k.sourceType { + t.Errorf("%s: UserSource is %v, want %s", label, typ, k.sourceType) + } + if id, _ := get(src, "$ID"); !sameID(id, wantID) { + t.Errorf("%s: UserSource $ID %v, want the targeting's own %v", label, id, wantID) + } + wantKeys := "$ID,$Type" + if k.valueKey != "" { + wantKeys += "," + k.valueKey + if v, _ := get(src, k.valueKey); v != k.value { + t.Errorf("%s: %s = %q, want %q", label, k.valueKey, v, k.value) + } + } + if got := strings.Join(keysOf(src), ","); got != wantKeys { + t.Errorf("%s: UserSource keys %s, want %s", label, got, wantKeys) + } + } + } + } +} + +// Group targeting is 11.2 (XPathGroupTargeting, MicroflowGroupTargeting, and +// the workflow groups they target): before, a user task targets users only. +// Refused naming the task, not written as a targeting the version reads as +// "XPath constraint is required". +func TestNormalize_RefusesGroupTargetingBelowMendix11_2(t *testing.T) { + group := func(typ, key, value string) bson.D { + return bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: typ}, {Key: key, Value: value}} + } + for _, g := range []bson.D{ + group("Workflows$XPathGroupTargeting", "XPathConstraint", "[Name = 'Reviewers']"), + group("Workflows$MicroflowGroupTargeting", "Microflow", "M.ACT_GetGroups"), + } { + for _, v := range []*version.ProjectVersion{pv(10, 24, 24), pv(11, 1, 0)} { + _, err := normalizeShape(t, v, workflowWithUserTasks(userTaskDoc("ByGroup", false, g))) + if err == nil || !strings.Contains(err.Error(), "ByGroup") || !strings.Contains(err.Error(), "11.2") { + t.Errorf("%s: group targeting on %s was not refused naming the task and 11.2: %v", g[1].Value, vlabel(v), err) + } + } + if _, err := normalizeShape(t, pv(11, 2, 0), workflowWithUserTasks(userTaskDoc("ByGroup", false, g))); err != nil { + t.Errorf("%s: refused on 11.2: %v", g[1].Value, err) + } + } +} From f9fcc87b498d35ed2c303fa5c6ef394f5662f1dd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:29:20 +0000 Subject: [PATCH 36/44] fix(canon): write a user task as a Workflows$UserTask below Mendix 10.12 Symptom (ako/mxcli#1771): on Mendix 9.x a workflow with a user task made the project unloadable. exec reported "Created workflow", then mx check 9.24.30 died with "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". Cause: mxcli writes Workflows$SingleUserTaskActivity and MultiUserTaskActivity. Mendix introduced both types in 10.12.0 and deleted Workflows$UserTask there (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared a written $Type with the project's version. Older shape, measured: Studio Pro 9.24.3 (the Marketplace's Multilevel Dynamic Workflow) stores {$ID, $Type, Annotation, AutoAssignSingleTargetUser, Caption, DueDate, Name, OnCreatedEvent, Outcomes, TaskDescription, TaskName, TaskPage, UserSource, UserTaskCompletion}. The completion (9.22) says which kind of task it is. A single-user task has a bare SingleInputCompletion. A multi-user task has a MultiInputCompletion holding its CompletionCriteria, its TargetUserInput and, from 10.2, AwaitAllUsers. Fix: canon.LowerToProjectShape rewrites the task as that UserTask below 10.12. The multi-user settings move into the completion, and the outcomes stay on the task, so the rule's outcome pointers still resolve. The completion's $ID is derived from the task's. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, majority and threshold below 10.0, a microflow rule below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. An empty BoundaryEvents list is dropped below 10.14; it is declared on the abstract UserTaskActivity, which the floor never saw. describe reads a 9.x multi-user task's completion, and the create-or-modify guard now sees its rule and participants too, so a rewrite no longer resets them silently. Measured on fresh apps, exec of the bug-test then mx check: - 9.24.30.51827: before TypeCacheUnknownTypeException, after 0 errors. mx convert 9.24.30 changes no key and no value. describe -> exec "Unchanged workflow". After mx convert 10.24.24, the user tasks are identical in keys and values to mxcli's direct 10.24 output. - 10.24.24 and 11.13.0: 0 errors. Control: stubbing lowerUserTaskType fails the three normalize tests on 9.24 with the 10.12 types in place. The read test and the guard test failed before their fixes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...2026-10-29-user-task-type-below-10-12.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 2 + .../bug-tests/1771-user-task-below-10-12.mdl | 80 ++++++ mdl/backend/modelsdk/workflow_read.go | 8 + .../modelsdk/workflow_usertask_shape_test.go | 54 ++++ mdl/executor/validate_workflow_rewrite.go | 14 +- .../validate_workflow_rewrite_1771_test.go | 54 ++++ modelsdk/canon/versionshape.go | 19 +- modelsdk/canon/versionshape_usertask.go | 149 +++++++++++ .../writer_version_shape_1771_type_test.go | 231 ++++++++++++++++++ 11 files changed, 608 insertions(+), 5 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json create mode 100644 mdl-examples/bug-tests/1771-user-task-below-10-12.mdl create mode 100644 mdl/executor/validate_workflow_rewrite_1771_test.go create mode 100644 modelsdk/mpr/writer_version_shape_1771_type_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json new file mode 100644 index 000000000..6421029ff --- /dev/null +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json @@ -0,0 +1 @@ +{"area": "modelsdk", "date": "2026-10-29", "symptom": "On Mendix 9.x a workflow with a user task made the project unloadable. exec reported 'Created workflow', then mx check 9.24.30 and Studio Pro died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity`. The same happened on 10.0–10.11 by the SDK's version data, which was not measured there", "cause": "userTaskToGen (and the ALTER mutator's inserts, through it) writes Workflows$SingleUserTaskActivity / MultiUserTaskActivity, TYPES introduced in 10.12.0, which also deleted Workflows$UserTask (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared the written $Type with the project's version. Before 10.12 the kind of task is the UserTask's UserTaskCompletion (9.22): SingleInputCompletion, or MultiInputCompletion {CompletionCriteria, TargetUserInput, AwaitAllUsers from 10.2}", "file": "`modelsdk/canon/versionshape_usertask.go` (lowerUserTaskType, dropTaskBoundaryEvents, userTaskActivitySince, completionCriteriaSince), `mdl/backend/modelsdk/workflow_read.go` (UserTask case reads the MultiInputCompletion), `mdl/executor/validate_workflow_rewrite.go` (rawMultiUserTaskSettings sees a 9.x multi-user task)", "insight": "**A type rename can also be a restructure.** Converting MultiUserTaskActivity down is more than a $Type swap. Three of its keys move into a nested MultiInputCompletion, and the outcomes stay on the task, so the CompletionCriteria's outcome pointers still resolve. Studio Pro 9.24.3's key set is the target: Multilevel Dynamic Workflow (Marketplace 221741) stores `$ID,$Type,Annotation,AutoAssignSingleTargetUser,Caption,DueDate,Name,OnCreatedEvent,Outcomes,TaskDescription,TaskName,TaskPage,UserSource,UserTaskCompletion`. **With no Studio Pro sample of a 9.x multi-user task, convert UP as the oracle.** The 9.24 output, after `mx convert -p` 10.24.24, has user tasks identical in keys and values to what mxcli writes on 10.24 directly, rule and participants included. **Every reader of the old shape needs the restructure too.** describe read a 9.x multi-user task as a single one, and the create-or-modify guard looked for the rule only on MultiUserTaskActivity, so a 9.24 rewrite would reset the rule silently. **Give an element the lowering ADDS a derived $ID** (completionID from the task's $ID), so the same task lowers to the same bytes. The boundary events on a user task (10.14, declared on the abstract UserTaskActivity) slipped past the floor the same way: an empty list is dropped below 10.14, and a boundary event is refused. Measured: bug-test on fresh apps. 9.24.30 before TypeCacheUnknownTypeException, after 0 errors. `mx convert -p` 9.24.30 changes no key and no value. describe → exec 'Unchanged workflow'. 10.24.24 and 11.13.0 give 0 errors. The CE2513 'Java 11' after converting up is in an untouched 9.24 app too. Control: stubbing lowerUserTaskType fails TestNormalize_UserTaskIsAWorkflowsUserTaskBelowMendix10_12, …MultiUserTask… and …RefusesWhatAnOlderUserTaskCannotHold with the 10.12 types on 9.24. The reader and guard tests failed before their fixes. Tests: those three, TestNormalize_UserTaskCompletionIDIsStable, TestWorkflowRead_UserTaskInTheMendix9Shape, TestWorkflowRewrite_SeesAMendix9MultiUserTasksSettings; mdl-examples/bug-tests/1771-user-task-below-10-12.mdl", "refs": ["#1771", "#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index eab978aac..f210553a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. - **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index 0af20a6fa..f5c8c1325 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -187,6 +187,7 @@ Some properties kept their meaning and changed how they are stored. mxcli builds | Any data source's `ForceFullObjects` | 10.8 | Not stored | | The `PersistentId` of a workflow, activity, outcome or boundary event | 10.21 | Not stored | | The end of a parallel split path (`Workflows$EndOfParallelSplitPathActivity`, a type introduced in 10.21) | 10.21 | Not stored: a path holds its activities and nothing more (Studio Pro 9.4) | +| A user task (`Workflows$SingleUserTaskActivity` / `MultiUserTaskActivity`, types introduced in 10.12) | 10.12 | A `Workflows$UserTask` whose `UserTaskCompletion` is a `SingleInputCompletion`, or a `MultiInputCompletion` holding the rule, the participants and, from 10.2, `AwaitAllUsers` (Studio Pro 9.24, [ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)) | | A user task's targeting (`targeting users xpath […]`, `targeting users microflow …`, none) | 11.2 | A `UserSource` (`XPathBasedUserSource`, `MicroflowBasedUserSource`, `EmptyUserSource`) holding the same value, instead of a `UserTargeting` ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)) | | A constant's value in a configuration (`alter settings constant … in configuration`) | 10.9 | A flat `Value` string on the override, instead of a `Settings$SharedValue` under `SharedOrPrivateValue` | | A page's flow call argument that is the enclosing object (`call microflow M.F(P = $currentObject)`, `call nanoflow …`) | 10.10 | Not stored: Studio Pro infers it. A page parameter, snippet parameter or widget argument is stored as before, as a `Variable` | @@ -199,6 +200,7 @@ What an older shape cannot hold is refused, naming the element, rather than drop - a private constant value in a configuration below 10.9; - a page's flow call argument that is any other expression (a literal, an attribute path) below 10.10; - a user task targeting groups (`targeting groups xpath …`, `targeting groups microflow …`) below 11.2, where a task targets users only ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); +- a multi-user task below 9.22, `decide by majority` or `decide by threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14 ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl b/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl new file mode 100644 index 000000000..bf4fe380e --- /dev/null +++ b/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl @@ -0,0 +1,80 @@ +mdl 1; +-- ako/mxcli#1771: on Mendix 9.x a workflow with a user task made the project +-- unloadable. `exec` reported "Created workflow", then mx check and Studio Pro +-- could not load the project at all: +-- +-- TypeCacheUnknownTypeException: The type cache does not contain a type with +-- qualified name Workflows$SingleUserTaskActivity. +-- +-- mxcli writes a user task as a Workflows$SingleUserTaskActivity or +-- MultiUserTaskActivity. Both types arrived in 10.12, which also deleted the +-- one before them (mendixmodelsdk). Before 10.12 a user task is a +-- Workflows$UserTask, and its UserTaskCompletion (9.22) says whether it is a +-- single-user or a multi-user task. Studio Pro 9.24.3 stores `{…, UserSource, +-- UserTaskCompletion: SingleInputCompletion}` (the Marketplace's Multilevel +-- Dynamic Workflow). A multi-user task's rule, participants and, from 10.2, +-- `await all users` sit in a MultiInputCompletion, and its outcomes stay on the +-- task. What the older type cannot hold is refused, naming the task: a rule +-- added later (majority and threshold in 10.0, microflow in 10.3), `await all +-- users` below 10.2, and a boundary event on a user task below 10.14. +-- +-- Measured on fresh apps (`mxcli new … --skip-build`), exec then +-- `mxcli docker check`: +-- 9.24.30.51827 before: TypeCacheUnknownTypeException. After: 0 errors. +-- `mx convert -p` 9.24.30 changes no key and no value. +-- describe → exec prints "Unchanged workflow". Converted up +-- to 10.24.24, the tasks are those mxcli writes on 10.24 +-- directly. +-- 10.24.24, 11.13.0 0 errors, with the same key sets as before the fix. + +create module Probe1771Ut; + +create persistent entity Probe1771Ut.Request ( Title: String(100) ); + +create microflow Probe1771Ut.ACT_GetUsers ($Workflow: System.Workflow, $Context: Probe1771Ut.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +create page Probe1771Ut.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txt1 (content: 'Task') +}; + +create workflow Probe1771Ut.WF_Tasks + parameter $WorkflowContext: Probe1771Ut.Request + display 'Tasks' +begin + user task Review 'Review' + page Probe1771Ut.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Approve' { } 'Reject' { }; + user task Handle 'Handle' + page Probe1771Ut.TaskPage + targeting users microflow Probe1771Ut.ACT_GetUsers + outcomes 'Done' { }; + multi user task Vote 'Vote' + page Probe1771Ut.TaskPage + targeting users xpath [Name = 'admin'] + participants 2 + decide by veto 'Reject' + outcomes 'Approve' { } 'Reject' { }; + multi user task Agree 'Agree' + page Probe1771Ut.TaskPage + targeting users microflow Probe1771Ut.ACT_GetUsers + outcomes 'Yes' { } 'No' { }; +end workflow; + +-- ALTER inserts a user task through the mutator, outside CREATE's encoder; the +-- lowering runs on every unit write, so it covers both. +alter workflow Probe1771Ut.WF_Tasks { + insert after Handle { user task Confirm 'Confirm' + page Probe1771Ut.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'OK' { }; } +}; diff --git a/mdl/backend/modelsdk/workflow_read.go b/mdl/backend/modelsdk/workflow_read.go index 5da51c1cd..31e79e9f3 100644 --- a/mdl/backend/modelsdk/workflow_read.go +++ b/mdl/backend/modelsdk/workflow_read.go @@ -192,6 +192,14 @@ func workflowActivityFromGen(el element.Element) workflows.WorkflowActivity { t.Page = taskPageName(a.TaskPage()) } t.Outcomes = userTaskOutcomesFromGen(a.OutcomesItems()) + // Below 10.12 a multi-user task is a UserTask whose completion holds + // what MultiUserTaskActivity holds itself from 10.12 (ako/mxcli#1771). + if mc, ok := a.UserTaskCompletion().(*genWf.MultiInputCompletion); ok && mc != nil { + t.IsMulti = true + t.AwaitAllUsers = mc.AwaitAllUsers() + t.CompletionCriteria = completionCriteriaFromGen(mc.CompletionCriteria(), t.Outcomes) + t.TargetUserInput = targetUserInputFromGen(mc.TargetUserInput()) + } return t case *genWf.SingleUserTaskActivity: t := &workflows.UserTask{ diff --git a/mdl/backend/modelsdk/workflow_usertask_shape_test.go b/mdl/backend/modelsdk/workflow_usertask_shape_test.go index 34d2ce89f..21914ac10 100644 --- a/mdl/backend/modelsdk/workflow_usertask_shape_test.go +++ b/mdl/backend/modelsdk/workflow_usertask_shape_test.go @@ -56,6 +56,60 @@ func storedUserTask(t *testing.T, act *workflows.UserTask, pv *types.ProjectVers return ut } +// Below 10.12 a user task is a Workflows$UserTask, and a multi-user task's +// rule, participants and `await all users` live in its UserTaskCompletion (a +// Workflows$MultiInputCompletion). Each reads back as the statement gave it, +// as on 10.12 and later. +func TestWorkflowRead_UserTaskInTheMendix9Shape(t *testing.T) { + outcomes := func() []*workflows.UserTaskOutcome { + return []*workflows.UserTaskOutcome{{Value: "Approve"}, {Value: "Reject"}} + } + for _, tc := range []struct { + name string + pv *types.ProjectVersion + multi bool + cc *workflows.CompletionCriteria + input *workflows.TargetUserInput + await bool + }{ + {"single", &types.ProjectVersion{MajorVersion: 9, MinorVersion: 24, PatchVersion: 30}, false, nil, nil, false}, + {"consensus", &types.ProjectVersion{MajorVersion: 9, MinorVersion: 24, PatchVersion: 30}, true, + &workflows.CompletionCriteria{Kind: "Consensus", FallbackOutcome: "Reject"}, &workflows.TargetUserInput{Kind: "All"}, false}, + {"veto, two participants", &types.ProjectVersion{MajorVersion: 9, MinorVersion: 24, PatchVersion: 30}, true, + &workflows.CompletionCriteria{Kind: "Veto", VetoOutcome: "Reject"}, &workflows.TargetUserInput{Kind: "Absolute", Amount: 2}, false}, + {"majority, await all", &types.ProjectVersion{MajorVersion: 10, MinorVersion: 6}, true, + &workflows.CompletionCriteria{Kind: "Majority", CompletionType: "Absolute", FallbackOutcome: "Approve"}, + &workflows.TargetUserInput{Kind: "Percentage", Percentage: 50}, true}, + {"control: 10.24", &types.ProjectVersion{MajorVersion: 10, MinorVersion: 24, PatchVersion: 24}, true, + &workflows.CompletionCriteria{Kind: "Veto", VetoOutcome: "Reject"}, &workflows.TargetUserInput{Kind: "Absolute", Amount: 2}, true}, + } { + ut := &workflows.UserTask{Page: "M.TaskPage", IsMulti: tc.multi, Outcomes: outcomes(), + UserSource: &workflows.XPathBasedUserSource{XPath: "[Name = 'admin']"}, + CompletionCriteria: tc.cc, TargetUserInput: tc.input, AwaitAllUsers: tc.await} + ut.Name, ut.Caption = "Review", "Review" + + got := storedUserTask(t, ut, tc.pv) + if got.IsMulti != tc.multi { + t.Errorf("%s: read back multi=%v, want %v", tc.name, got.IsMulti, tc.multi) + } + if !reflect.DeepEqual(got.UserSource, ut.UserSource) { + t.Errorf("%s: targeting read back as %#v", tc.name, got.UserSource) + } + if !tc.multi { + continue + } + if got.CompletionCriteria == nil || *got.CompletionCriteria != *tc.cc { + t.Errorf("%s: rule read back as %+v, want %+v", tc.name, got.CompletionCriteria, tc.cc) + } + if got.TargetUserInput == nil || *got.TargetUserInput != *tc.input { + t.Errorf("%s: participants read back as %+v, want %+v", tc.name, got.TargetUserInput, tc.input) + } + if got.AwaitAllUsers != tc.await { + t.Errorf("%s: await all users read back as %v, want %v", tc.name, got.AwaitAllUsers, tc.await) + } + } +} + func TestWorkflowRead_UserTaskTargetingInTheMendix10Shape(t *testing.T) { for _, tc := range []struct { name string diff --git a/mdl/executor/validate_workflow_rewrite.go b/mdl/executor/validate_workflow_rewrite.go index 6cd7d63ee..ad80144ce 100644 --- a/mdl/executor/validate_workflow_rewrite.go +++ b/mdl/executor/validate_workflow_rewrite.go @@ -347,7 +347,19 @@ func rawAwaitsAllUsers(d map[string]any) bool { // resets when the statement omits it. func rawMultiUserTaskSettings(raw map[string]any) (rules, participants, await []string) { walkRawDocs(raw, func(d map[string]any) { - if t, _ := d["$Type"].(string); t != "Workflows$MultiUserTaskActivity" { + switch t, _ := d["$Type"].(string); t { + case "Workflows$MultiUserTaskActivity": + case "Workflows$UserTask": + // Below 10.12 a multi-user task is a UserTask, and its settings live + // on its MultiInputCompletion (ako/mxcli#1771). + c, _ := d["UserTaskCompletion"].(map[string]any) + if ct, _ := c["$Type"].(string); ct != "Workflows$MultiInputCompletion" { + return + } + d = map[string]any{"Name": d["Name"], "Outcomes": d["Outcomes"], + "CompletionCriteria": c["CompletionCriteria"], "TargetUserInput": c["TargetUserInput"], + "AwaitAllUsers": c["AwaitAllUsers"]} + default: return } if r := rawCompletionRule(d); r != "" { diff --git a/mdl/executor/validate_workflow_rewrite_1771_test.go b/mdl/executor/validate_workflow_rewrite_1771_test.go new file mode 100644 index 000000000..e2c653b1b --- /dev/null +++ b/mdl/executor/validate_workflow_rewrite_1771_test.go @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// mendix9MultiUserTaskWorkflow is a workflow as a Mendix 9.x–10.11 project +// stores it (ako/mxcli#1771): a multi-user task is a Workflows$UserTask whose +// UserTaskCompletion, a MultiInputCompletion, holds its rule and participants; +// a single-user task's is a SingleInputCompletion. +func mendix9MultiUserTaskWorkflow() map[string]any { + first := map[string]any{"$ID": "outcome-fast", "$Type": "Workflows$UserTaskOutcome", "Value": "Fast"} + second := map[string]any{"$ID": "outcome-furious", "$Type": "Workflows$UserTaskOutcome", "Value": "Furious"} + return map[string]any{ + "$Type": "Workflows$Workflow", + "Flow": map[string]any{"$Type": "Workflows$Flow", "Activities": []any{ + 3, + map[string]any{"$Type": "Workflows$UserTask", "Name": "userTask1", + "OnCreatedEvent": map[string]any{"$Type": "Workflows$NoEvent"}, + "UserTaskCompletion": map[string]any{"$Type": "Workflows$SingleInputCompletion"}}, + map[string]any{"$Type": "Workflows$UserTask", "Name": "userTask2", + "OnCreatedEvent": map[string]any{"$Type": "Workflows$NoEvent"}, + "Outcomes": []any{3, first, second}, + "UserTaskCompletion": map[string]any{"$Type": "Workflows$MultiInputCompletion", + "CompletionCriteria": map[string]any{"$Type": "Workflows$VetoCompletionCriteria", "VetoOutcomePointer": "outcome-furious"}, + "TargetUserInput": map[string]any{"$Type": "Workflows$AbsoluteAmountUserInput", "Amount": 2}}}, + }}, + } +} + +// The rewrite guard refuses a statement that would reset a stored multi-user +// task's rule or participants. Below 10.12 they are stored on the task's +// UserTaskCompletion, where the guard did not look, so a rewrite on 9.24 reset +// both to the defaults silently. +func TestWorkflowRewrite_SeesAMendix9MultiUserTasksSettings(t *testing.T) { + stmt := parseWorkflowStmt(t, wfRewriteTwoTasks) + err := checkNoDroppedWorkflowConstructs(rawWorkflowCtx(t, mendix9MultiUserTaskWorkflow()), "wf1", "M.W", stmt) + if err == nil || !strings.Contains(err.Error(), "veto") { + t.Fatalf("a rewrite dropping a 9.x multi-user task's veto rule must be refused naming it, got %v", err) + } + + // Control: restating the rule and the participants loses nothing. + restated := parseWorkflowStmt(t, `create or modify workflow M.W parameter $C: M.Ctx +begin + user task userTask1 'User Task' page M.P outcomes 'Good' { } 'Bad' { }; + multi user task userTask2 'Multi' page M.P participants 2 decide by veto 'Furious' outcomes 'Fast' { } 'Furious' { }; +end workflow;`) + if err := checkNoDroppedWorkflowConstructs(rawWorkflowCtx(t, mendix9MultiUserTaskWorkflow()), "wf1", "M.W", restated); err != nil { + t.Fatalf("a rewrite that restates the rule and participants must be allowed, got %v", err) + } +} diff --git a/modelsdk/canon/versionshape.go b/modelsdk/canon/versionshape.go index 4b51ae071..fbf5097cd 100644 --- a/modelsdk/canon/versionshape.go +++ b/modelsdk/canon/versionshape.go @@ -62,6 +62,9 @@ import ( // - Workflows$ user task targeting: UserTargeting from 11.2.0, declared on // the abstract UserTaskActivity; before, UserSource (ako/mxcli#1771). See // lowerUserTargeting. +// - Workflows$SingleUserTaskActivity and MultiUserTaskActivity, TYPES +// introduced in 10.12.0; before, a Workflows$UserTask whose +// UserTaskCompletion says which (ako/mxcli#1771). See lowerUserTaskType. // // It runs on every unit write, before the floor, so no writer needs to know // these shapes. What the older shape cannot hold — a design property value @@ -83,15 +86,19 @@ func LowerToProjectShape(contents []byte, pv *version.Version) ([]byte, error) { variableArgs: !PropertyDeclared("Forms$MicroflowParameterMapping", "Expression", *pv), unmarkedPaths: pv.Compare(endOfParallelSplitPathSince) < 0, userSource: !PropertyDeclared("Workflows$UserTaskActivity", "UserTargeting", *pv), + taskBoundaries: !PropertyDeclared("Workflows$UserTaskActivity", "BoundaryEvents", *pv), + userTaskType: pv.Compare(userTaskActivitySince) < 0, } if !l.flatDesignProps && !l.boolEditable && !l.textPlaceholder && !l.namedParameter && !l.noFullObjects && - !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths && !l.userSource { + !l.noPersistentID && !l.flatConstant && !l.variableArgs && !l.unmarkedPaths && !l.userSource && + !l.taskBoundaries && !l.userTaskType { return contents, nil } var doc bson.D if err := bson.Unmarshal(contents, &doc); err != nil { return nil, err } + l.root = doc out, err := l.walk(doc) if err != nil { return nil, err @@ -107,8 +114,12 @@ type shapeLowering struct { flatDesignProps, boolEditable, textPlaceholder bool namedParameter, noFullObjects, noPersistentID bool flatConstant, variableArgs, unmarkedPaths bool - userSource bool + userSource, taskBoundaries, userTaskType bool changed bool + // root is the unit, and used the $IDs it holds, for an element the + // lowering adds (see lowerUserTaskType). + root bson.D + used map[string]bool } // endOfParallelSplitPathSince is the version that introduced the type @@ -150,8 +161,8 @@ func (l *shapeLowering) walk(v any) (any, error) { if x, err = l.lowerConstantValue(x); err != nil { return nil, err } - case l.userSource && userTaskTypes[typ]: - if x, err = l.lowerUserTargeting(x); err != nil { + case userTaskTypes[typ]: + if x, err = l.lowerUserTask(x, typ); err != nil { return nil, err } } diff --git a/modelsdk/canon/versionshape_usertask.go b/modelsdk/canon/versionshape_usertask.go index 8ce65d694..d436e4e4b 100644 --- a/modelsdk/canon/versionshape_usertask.go +++ b/modelsdk/canon/versionshape_usertask.go @@ -39,6 +39,155 @@ var userSourceOf = map[string]string{ // a user task with no targeting (mendixmodelsdk 4.117.0: introduced "9.10.0"). var emptyUserSourceSince = version.Version{Major: 9, Minor: 10} +// userTaskActivitySince is the version that introduced the types +// Workflows$SingleUserTaskActivity and MultiUserTaskActivity, and deleted +// Workflows$UserTask (mendixmodelsdk 4.117.0). Written down here because the +// generated version data holds property versions only, never a type's own. +var userTaskActivitySince = version.Version{Major: 10, Minor: 12} + +// completionCriteriaSince holds the multi-user task rules introduced after the +// multi-user task itself (9.22.0, with consensus and veto): mendixmodelsdk +// 4.117.0 StructureVersionInfo. +var completionCriteriaSince = map[string]version.Version{ + "Workflows$MajorityCompletionCriteria": {Major: 10}, + "Workflows$ThresholdCompletionCriteria": {Major: 10}, + "Workflows$MicroflowCompletionCriteria": {Major: 10, Minor: 3}, +} + +// completionRule names a completion criteria $Type as the statement that +// writes it does. +var completionRule = map[string]string{ + "Workflows$MajorityCompletionCriteria": "decide by majority", + "Workflows$ThresholdCompletionCriteria": "decide by threshold", + "Workflows$MicroflowCompletionCriteria": "decide by microflow", +} + +// lowerUserTask writes a user task in the shape the project's version stores: +// its targeting, its boundary events and, below 10.12, its type. +func (l *shapeLowering) lowerUserTask(task bson.D, typ string) (bson.D, error) { + var err error + if l.userSource { + if task, err = l.lowerUserTargeting(task); err != nil { + return nil, err + } + } + if l.taskBoundaries { + if task, err = l.dropTaskBoundaryEvents(task); err != nil { + return nil, err + } + } + if l.userTaskType && typ != "Workflows$UserTask" { + return l.lowerUserTaskType(task, typ) + } + return task, nil +} + +// dropTaskBoundaryEvents removes a user task's BoundaryEvents below 10.14, +// where a user task declares none (mendixmodelsdk: UserTaskActivity +// .boundaryEvents introduced 10.14.0; a Workflows$UserTask has none at all). +// The codec writes an empty list on every task, which is dropped; a boundary +// event is refused, naming the task. The floor never saw the key: it is +// declared on the abstract UserTaskActivity. +func (l *shapeLowering) dropTaskBoundaryEvents(task bson.D) (bson.D, error) { + i := shapeIndex(task, "BoundaryEvents") + if i < 0 { + return task, nil + } + if !isEmptyValue(task[i].Value) { + name, _ := shapeString(task, "Name") + return nil, fmt.Errorf("user task %q has a boundary event, which Mendix introduced on user tasks in 10.14 "+ + "(this project is %s)", name, l.pv) + } + l.changed = true + return append(append(bson.D{}, task[:i]...), task[i+1:]...), nil +} + +// lowerUserTaskType rewrites a Workflows$SingleUserTaskActivity or +// MultiUserTaskActivity as the Workflows$UserTask a version before 10.12 +// stores. Studio Pro 9.24.3 stores `{…, UserSource, UserTaskCompletion}` +// (Multilevel Dynamic Workflow, Marketplace 221741), and the completion says +// which kind of task it is (9.22.0): +// +// - a single-user task: a bare Workflows$SingleInputCompletion; +// - a multi-user task: a Workflows$MultiInputCompletion holding what the +// MultiUserTaskActivity holds itself from 10.12: CompletionCriteria (its +// outcome pointers unchanged, the outcomes stay on the task), +// TargetUserInput and, from 10.2.0, AwaitAllUsers. +// +// A 9.24.30 project holding the 10.12 types does not load +// (TypeCacheUnknownTypeException). What the older type cannot hold is refused, +// naming the task: a multi-user task below 9.22, a rule introduced later +// (majority and threshold 10.0, microflow 10.3), and `await all users` below +// 10.2. The completion's $ID is derived from the task's, so the same task +// lowers to the same bytes. +func (l *shapeLowering) lowerUserTaskType(task bson.D, typ string) (bson.D, error) { + name, _ := shapeString(task, "Name") + hasCompletion := PropertyDeclared("Workflows$UserTask", "UserTaskCompletion", l.pv) + completion := bson.D{{Key: "$ID", Value: l.derivedID(task, "UserTaskCompletion")}} + if typ == "Workflows$MultiUserTaskActivity" { + if !hasCompletion { + return nil, fmt.Errorf("multi user task %q: Mendix introduced multi-user tasks in 9.22 (this project is %s)", + name, l.pv) + } + completion = append(completion, bson.E{Key: "$Type", Value: "Workflows$MultiInputCompletion"}) + var await, criteria, input any + task, await = takeKey(task, "AwaitAllUsers") + task, criteria = takeKey(task, "CompletionCriteria") + task, input = takeKey(task, "TargetUserInput") + if PropertyDeclared("Workflows$MultiInputCompletion", "AwaitAllUsers", l.pv) { + completion = append(completion, bson.E{Key: "AwaitAllUsers", Value: await == true}) + } else if await == true { + return nil, fmt.Errorf("multi user task %q waits for all users, which Mendix introduced in 10.2 (this project is %s)", + name, l.pv) + } + criteriaDoc, _ := criteria.(bson.D) + ct, _ := shapeString(criteriaDoc, "$Type") + if since, ok := completionCriteriaSince[ct]; ok && l.pv.Compare(since) < 0 { + return nil, fmt.Errorf("multi user task %q is decided by %s, which Mendix introduced in %d.%d (this project is %s)", + name, completionRule[ct], since.Major, since.Minor, l.pv) + } + completion = append(completion, + bson.E{Key: "CompletionCriteria", Value: criteria}, + bson.E{Key: "TargetUserInput", Value: input}) + } else { + completion = append(completion, bson.E{Key: "$Type", Value: "Workflows$SingleInputCompletion"}) + } + task[shapeIndex(task, "$Type")].Value = "Workflows$UserTask" + l.changed = true + if !hasCompletion { + return task, nil + } + // In Studio Pro's place: after the user source, its neighbour in the + // alphabetical order Studio Pro writes. + at := len(task) + if i := shapeIndex(task, "UserSource"); i >= 0 { + at = i + 1 + } + out := append(append(bson.D{}, task[:at]...), bson.E{Key: "UserTaskCompletion", Value: completion}) + return append(out, task[at:]...), nil +} + +// takeKey removes key from d, returning d without it and the value it held. +func takeKey(d bson.D, key string) (bson.D, any) { + i := shapeIndex(d, key) + if i < 0 { + return d, nil + } + v := d[i].Value + return append(append(bson.D{}, d[:i]...), d[i+1:]...), v +} + +// derivedID is an $ID for an element the lowering adds to owner, derived from +// the owner's $ID and the key it is added under, as CompletePropertySets +// derives the ones it adds: the same owner always gets the same id, and one +// the unit already holds is stepped past. +func (l *shapeLowering) derivedID(owner bson.D, key string) bson.Binary { + if l.used == nil { + l.used = elementIDs(l.root) + } + return completionID(elementIDBytes(owner), "LowerToProjectShape/"+key, l.used) +} + // lowerUserTargeting replaces a user task's UserTargeting with the UserSource a // version before 11.2 stores, in its place. mxcli writes UserTargeting on every // version, and the floor never saw it: it is declared on the abstract diff --git a/modelsdk/mpr/writer_version_shape_1771_type_test.go b/modelsdk/mpr/writer_version_shape_1771_type_test.go new file mode 100644 index 000000000..29c747732 --- /dev/null +++ b/modelsdk/mpr/writer_version_shape_1771_type_test.go @@ -0,0 +1,231 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "bytes" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/modelsdk/mpr/version" +) + +// studioPro924UserTask is the key set Studio Pro 9.24.3 stores on a user task +// (Multilevel Dynamic Workflow, Marketplace 221741). Employee Onboarding +// (9.1.1), converted by mx convert 9.24.30, stores the same set. +const studioPro924UserTask = "$ID,$Type,Annotation,AutoAssignSingleTargetUser,Caption,DueDate,Name,OnCreatedEvent," + + "Outcomes,TaskDescription,TaskName,TaskPage,UserSource,UserTaskCompletion" + +// Below 10.12 a user task is a Workflows$UserTask. mendixmodelsdk deletes +// UserTask and introduces SingleUserTaskActivity and MultiUserTaskActivity in +// 10.12.0. Whether the task is single- or multi-user is its UserTaskCompletion +// (9.22.0): a SingleInputCompletion, or a MultiInputCompletion holding the +// completion criteria, the participants and, from 10.2.0, AwaitAllUsers. +// mxcli wrote the 10.12 types on every version, and 9.24.30 then did not load +// the project (TypeCacheUnknownTypeException … Workflows$SingleUserTaskActivity). +func TestNormalize_UserTaskIsAWorkflowsUserTaskBelowMendix10_12(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + userTask bool + }{ + {pv(9, 24, 30), true}, + {pv(10, 6, 0), true}, + {pv(10, 11, 0), true}, + {pv(10, 12, 0), false}, + {pv(10, 24, 24), false}, // control + } { + d, err := normalizeShape(t, tt.v, workflowWithUserTasks(userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")))) + if err != nil { + t.Errorf("%s: refused: %v", vlabel(tt.v), err) + continue + } + task := findNamed(d, "Single") + typ, _ := get(task, "$Type") + if !tt.userTask { + if typ != "Workflows$SingleUserTaskActivity" { + t.Errorf("%s: $Type %v, want Workflows$SingleUserTaskActivity", vlabel(tt.v), typ) + } + if _, ok := get(task, "UserTaskCompletion"); ok { + t.Errorf("%s: a SingleUserTaskActivity holds a UserTaskCompletion", vlabel(tt.v)) + } + continue + } + if typ != "Workflows$UserTask" { + t.Errorf("%s: $Type %v, want Workflows$UserTask", vlabel(tt.v), typ) + } + if got := sortedKeys(task); got != studioPro924UserTask { + t.Errorf("%s: user task keys\n %s\nwant Studio Pro 9.24's\n %s", vlabel(tt.v), got, studioPro924UserTask) + } + completion, _ := get(task, "UserTaskCompletion") + cd, _ := completion.(bson.D) + if ct, _ := get(cd, "$Type"); ct != "Workflows$SingleInputCompletion" || len(cd) != 2 { + t.Errorf("%s: UserTaskCompletion %v, want a bare Workflows$SingleInputCompletion", vlabel(tt.v), cd) + } + if id, _ := get(cd, "$ID"); !nonZeroID(id) { + t.Errorf("%s: UserTaskCompletion has no $ID: %v", vlabel(tt.v), id) + } + } +} + +// The completion's $ID is derived from the task's, so writing the same task +// twice produces the same bytes. +func TestNormalize_UserTaskCompletionIDIsStable(t *testing.T) { + doc := workflowWithUserTasks(userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']"))) + a, err := normalizeShape(t, pv(9, 24, 30), doc) + if err != nil { + t.Fatal(err) + } + b, err := normalizeShape(t, pv(9, 24, 30), doc) + if err != nil { + t.Fatal(err) + } + ida, _ := get(find(a, "Workflows$SingleInputCompletion"), "$ID") + idb, _ := get(find(b, "Workflows$SingleInputCompletion"), "$ID") + if !sameID(ida, idb) { + t.Errorf("the completion's $ID differs between two writes of the same task: %v, %v", ida, idb) + } +} + +func nonZeroID(v any) bool { + b, ok := v.(bson.Binary) + return ok && len(b.Data) == 16 && !bytes.Equal(b.Data, make([]byte, 16)) +} + +// A multi-user task below 10.12 is a Workflows$UserTask whose +// UserTaskCompletion is a Workflows$MultiInputCompletion. From 10.12 the +// MultiUserTaskActivity holds the same things itself: CompletionCriteria (its +// outcome pointers unchanged), TargetUserInput and, from 10.2, AwaitAllUsers. +func TestNormalize_MultiUserTaskBelowMendix10_12(t *testing.T) { + for _, tt := range []struct { + v *version.ProjectVersion + completion string // keys of the MultiInputCompletion; "" = a MultiUserTaskActivity + }{ + {pv(9, 24, 30), "$ID,$Type,CompletionCriteria,TargetUserInput"}, + {pv(10, 1, 0), "$ID,$Type,CompletionCriteria,TargetUserInput"}, + {pv(10, 2, 0), "$ID,$Type,AwaitAllUsers,CompletionCriteria,TargetUserInput"}, + {pv(10, 11, 0), "$ID,$Type,AwaitAllUsers,CompletionCriteria,TargetUserInput"}, + {pv(10, 12, 0), ""}, + {pv(10, 24, 24), ""}, // control + } { + doc := userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']")) + criteria, _ := get(doc, "CompletionCriteria") + input, _ := get(doc, "TargetUserInput") + d, err := normalizeShape(t, tt.v, workflowWithUserTasks(doc)) + if err != nil { + t.Errorf("%s: refused: %v", vlabel(tt.v), err) + continue + } + task := findNamed(d, "Multi") + typ, _ := get(task, "$Type") + if tt.completion == "" { + if typ != "Workflows$MultiUserTaskActivity" { + t.Errorf("%s: $Type %v, want Workflows$MultiUserTaskActivity", vlabel(tt.v), typ) + } + continue + } + if typ != "Workflows$UserTask" { + t.Errorf("%s: $Type %v, want Workflows$UserTask", vlabel(tt.v), typ) + } + if got := sortedKeys(task); got != studioPro924UserTask { + t.Errorf("%s: user task keys\n %s\nwant\n %s", vlabel(tt.v), got, studioPro924UserTask) + } + completion, _ := get(task, "UserTaskCompletion") + cd, _ := completion.(bson.D) + if ct, _ := get(cd, "$Type"); ct != "Workflows$MultiInputCompletion" { + t.Errorf("%s: UserTaskCompletion is %v, want Workflows$MultiInputCompletion", vlabel(tt.v), ct) + continue + } + if got := strings.Join(keysOf(cd), ","); got != tt.completion { + t.Errorf("%s: MultiInputCompletion keys %s, want %s", vlabel(tt.v), got, tt.completion) + } + gotCriteria, _ := get(cd, "CompletionCriteria") + gotInput, _ := get(cd, "TargetUserInput") + if !sameValue(gotCriteria, criteria) || !sameValue(gotInput, input) { + t.Errorf("%s: criteria %v and participants %v, want the task's own %v and %v", vlabel(tt.v), gotCriteria, gotInput, criteria, input) + } + } +} + +// What a Workflows$UserTask cannot hold is refused, naming the task: +// - a multi-user task below 9.22; +// - a rule Mendix had not introduced yet: majority and threshold are 10.0, +// a microflow rule is 10.3; +// - `await all users` below 10.2; +// - a boundary event on a user task below 10.14. +// +// An empty BoundaryEvents list is the codec's default, and it is dropped. +func TestNormalize_RefusesWhatAnOlderUserTaskCannotHold(t *testing.T) { + withCriteria := func(typ string, extra ...bson.E) bson.D { + doc := userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']")) + cc := bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: typ}} + cc = append(cc, extra...) + doc[indexOf(doc, "CompletionCriteria")].Value = cc + return doc + } + awaiting := userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']")) + awaiting[indexOf(awaiting, "AwaitAllUsers")].Value = true + withBoundary := userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")) + withBoundary[indexOf(withBoundary, "BoundaryEvents")].Value = bson.A{int32(2), + bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$InterruptingTimerBoundaryEvent"}, {Key: "Caption", Value: "Late"}}} + for _, tt := range []struct { + name string + v *version.ProjectVersion + doc bson.D + want string + }{ + {"multi-user task", pv(9, 21, 0), userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']")), "9.22"}, + {"await all users", pv(9, 24, 30), awaiting, "10.2"}, + {"majority", pv(9, 24, 30), withCriteria("Workflows$MajorityCompletionCriteria", bson.E{Key: "CompletionType", Value: "Absolute"}), "10.0"}, + {"threshold", pv(9, 24, 30), withCriteria("Workflows$ThresholdCompletionCriteria", bson.E{Key: "Threshold", Value: int32(2)}), "10.0"}, + {"microflow rule", pv(10, 2, 0), withCriteria("Workflows$MicroflowCompletionCriteria", bson.E{Key: "Microflow", Value: "M.Decide"}), "10.3"}, + {"boundary event 9.24", pv(9, 24, 30), withBoundary, "10.14"}, + {"boundary event 10.13", pv(10, 13, 0), withBoundary, "10.14"}, + } { + name, _ := get(tt.doc, "Name") + _, err := normalizeShape(t, tt.v, workflowWithUserTasks(tt.doc)) + if err == nil || !strings.Contains(err.Error(), name.(string)) || !strings.Contains(err.Error(), tt.want) { + t.Errorf("%s on %s was not refused naming the task and %s: %v", tt.name, vlabel(tt.v), tt.want, err) + } + } + // Control: each is written where Mendix has it. + for _, tt := range []struct { + v *version.ProjectVersion + doc bson.D + }{ + {pv(9, 22, 0), userTaskDoc("Multi", true, xpathUserTargeting("[Name = 'admin']"))}, + {pv(10, 2, 0), awaiting}, + {pv(10, 0, 0), withCriteria("Workflows$MajorityCompletionCriteria", bson.E{Key: "CompletionType", Value: "Absolute"})}, + {pv(10, 3, 0), withCriteria("Workflows$MicroflowCompletionCriteria", bson.E{Key: "Microflow", Value: "M.Decide"})}, + {pv(10, 14, 0), withBoundary}, + } { + if _, err := normalizeShape(t, tt.v, workflowWithUserTasks(tt.doc)); err != nil { + t.Errorf("control %s: refused: %v", vlabel(tt.v), err) + } + } + // The empty list the codec writes on every task is dropped below 10.14. + d, err := normalizeShape(t, pv(10, 13, 0), workflowWithUserTasks(userTaskDoc("Single", false, xpathUserTargeting("[Name = 'admin']")))) + if err != nil { + t.Fatal(err) + } + if _, ok := get(findNamed(d, "Single"), "BoundaryEvents"); ok { + t.Errorf("10.13: an empty BoundaryEvents is kept on a user task, which declares none before 10.14") + } +} + +func indexOf(d bson.D, key string) int { + for i, e := range d { + if e.Key == key { + return i + } + } + panic("no " + key) +} + +// sameValue compares two stored values by their BSON bytes. +func sameValue(a, b any) bool { + x, err1 := bson.Marshal(bson.D{{Key: "v", Value: a}}) + y, err2 := bson.Marshal(bson.D{{Key: "v", Value: b}}) + return err1 == nil && err2 == nil && bytes.Equal(x, y) +} From 3c69d95118c71fd3238b43e4cb0a6f918228cdff Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:38:05 +0000 Subject: [PATCH 37/44] fix(wfmutator): set a user task's targeting whichever key stores it Symptom (ako/mxcli#1771): on Mendix 9.x to 11.1, `alter workflow M.W { set (Targeting: xpath [...]) on Task; }`, and the deprecated `set activity 'Task' targeting ...` form, reported "Altered workflow" and changed nothing. describe still showed the old targeting, and mx check stayed clean. Cause: SetActivityProperty set UserTargeting with bsonnav.DSet, which only replaces a key that already exists. Below 11.2 a stored user task holds its targeting as UserSource, so no key matched and nothing changed. Fix: setUserTargeting replaces UserTargeting or UserSource in place, or adds the targeting when the task has neither. It writes the current shape; canon.LowerToProjectShape stores it as the project's version does, as it does for every other writer. Measured on fresh apps, the bug-test exec'd then described and checked: - 9.24.30.51827 and 10.24.24.119653: with the committed mutator (a scratch build), "Altered workflow" x2 and `[Name = 'admin']` kept. With the fix, `[Name = 'reviewer']` and ACT_GetReviewers, 0 errors, and the version's key set (Workflows$UserTask with UserSource on 9.24). - 11.13.0: the new targeting both ways, 0 errors. Control: TestWorkflowMutator_SetTargetingReplacesAStoredUserSource and ...AddsAMissingTargeting failed before the fix. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...9-alter-set-targeting-noop-below-11-2.json | 1 + CHANGELOG.md | 1 + .../1771-alter-user-task-targeting.mdl | 61 ++++++++++++++ mdl/backend/wfmutator/mutator.go | 26 ++++-- .../wfmutator/set_targeting_1771_test.go | 79 +++++++++++++++++++ .../writer_version_shape_1771_type_test.go | 28 +++++++ 6 files changed, 190 insertions(+), 6 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-backend/2026-10-29-alter-set-targeting-noop-below-11-2.json create mode 100644 mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl create mode 100644 mdl/backend/wfmutator/set_targeting_1771_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend/2026-10-29-alter-set-targeting-noop-below-11-2.json b/.claude/skills/fix-issue/findings/mdl-backend/2026-10-29-alter-set-targeting-noop-below-11-2.json new file mode 100644 index 000000000..bc94ff214 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-backend/2026-10-29-alter-set-targeting-noop-below-11-2.json @@ -0,0 +1 @@ +{"area": "mdl/backend", "date": "2026-10-29", "symptom": "On Mendix 9.x to 11.1, `alter workflow M.W { set (Targeting: xpath […]) on Task; }`, and the deprecated `set activity 'Task' targeting …` form, printed 'Altered workflow' and changed nothing. describe showed the old targeting, and mx check stayed clean, because the old targeting was still valid", "cause": "Mutator.SetActivityProperty set the targeting with bsonnav.DSet(actDoc, \"UserTargeting\", …), and DSet only replaces a key that already exists. Below 11.2 a stored user task holds its targeting as UserSource, so no key matched, nothing changed, and the op returned nil", "file": "`mdl/backend/wfmutator/mutator.go` (setUserTargeting: replaces UserTargeting or UserSource in place, else appends; the write lowers it per version in canon.LowerToProjectShape)", "insight": "**A setter that only replaces an existing key is a silent no-op on a document in another version's shape.** bsonnav.DSet reports false, and every caller here ignored it. When a storage key is renamed at a version (ako/mxcli#1771: UserSource → UserTargeting at 11.2), grep the raw mutators for the new key: each one that DSets it is broken on the older versions, and the success message hides it. Have the mutator write the CURRENT shape in place of whichever key holds the fact, and let canon.LowerToProjectShape convert it. A second copy of the old shape in the mutator would be a second mapping to keep in step. **The control needs a pre-fix binary for the exact bug-test**: rebuilding with the committed file swapped in (go build -o scratch) showed 'Altered workflow' x2 with `[Name = 'admin']` kept on 9.24.30 and 10.24.24. Measured after the fix: 9.24.30 and 10.24.24 describe the new targeting, with 0 errors and the version's key set (Workflows$UserTask with UserSource on 9.24). 11.13.0 is unchanged. Tests: TestWorkflowMutator_SetTargetingReplacesAStoredUserSource and …AddsAMissingTargeting (both failed before the fix), TestNormalize_EditedMendix9UserTaskTargeting; mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl", "refs": ["#1771"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index f210553a8..5b5f0ce2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **`alter workflow … set (Targeting: …)` changes a user task's targeting on Mendix 9 to 11.1** (ako/mxcli#1771) — it reported "Altered workflow" and changed nothing. Before 11.2 a user task stores its targeting under another key, and the edit only replaced a key that was already there. The edit now replaces the stored targeting, whichever key holds it, and the write stores it in the version's shape. Measured on 9.24.30 and 10.24.24: `describe` shows the new targeting, with 0 errors. 11.2 and later are unchanged. - **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. - **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. diff --git a/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl b/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl new file mode 100644 index 000000000..dff335a7a --- /dev/null +++ b/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl @@ -0,0 +1,61 @@ +mdl 1; +-- ako/mxcli#1771: on Mendix 9.x to 11.1, `alter workflow … { set (Targeting: …) +-- on … }` reported "Altered workflow" and changed nothing. describe +-- printed the old targeting afterwards, and mx check stayed clean, because +-- the old targeting was still valid. +-- +-- Below 11.2 a stored user task holds its targeting as UserSource. The ALTER +-- mutator set UserTargeting, the 11.2 key, and only where the task already had +-- one, so on those versions it found none and did nothing. It now replaces the +-- stored targeting, whichever key holds it. The write then stores it in the +-- version's shape (see 1771-user-task-targeting-below-11-2.mdl). +-- +-- Measured on fresh apps (`mxcli new … --skip-build`): exec, then describe, +-- then `mxcli docker check`. +-- 9.24.30.51827, 10.24.24.119653 before: the old targeting was kept. After: +-- `[Name = 'reviewer']` and ACT_GetReviewers. 0 errors, with +-- the version's key set (Workflows$UserTask on 9.24). +-- 11.13.0 the new targeting both before and after, 0 errors. + +create module Probe1771At; + +create persistent entity Probe1771At.Request ( Title: String(100) ); + +create microflow Probe1771At.ACT_GetUsers ($Workflow: System.Workflow, $Context: Probe1771At.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +create microflow Probe1771At.ACT_GetReviewers ($Workflow: System.Workflow, $Context: Probe1771At.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User where Name = 'reviewer'; + return $Users; +end; + +create page Probe1771At.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txt1 (content: 'Task') +}; + +create workflow Probe1771At.WF_Edit + parameter $WorkflowContext: Probe1771At.Request + display 'Edit' +begin + user task Review 'Review' + page Probe1771At.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Done' { }; + user task Handle 'Handle' + page Probe1771At.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Done' { }; +end workflow; + +alter workflow Probe1771At.WF_Edit { set (Targeting: xpath [Name = 'reviewer']) on Review; }; +alter workflow Probe1771At.WF_Edit { set (Targeting: microflow Probe1771At.ACT_GetReviewers) on Handle; }; diff --git a/mdl/backend/wfmutator/mutator.go b/mdl/backend/wfmutator/mutator.go index 00a04e75a..2dc409d3f 100644 --- a/mdl/backend/wfmutator/mutator.go +++ b/mdl/backend/wfmutator/mutator.go @@ -313,21 +313,19 @@ func (m *Mutator) SetActivityProperty(activityRef string, atPos int, prop string return nil case "targeting_microflow": - userTargeting := bson.D{ + m.setUserTargeting(actDoc, bson.D{ {Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, {Key: "$Type", Value: "Workflows$MicroflowUserTargeting"}, {Key: "Microflow", Value: value}, - } - bsonnav.DSet(actDoc, "UserTargeting", userTargeting) + }) return nil case "targeting_xpath": - userTargeting := bson.D{ + m.setUserTargeting(actDoc, bson.D{ {Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, {Key: "$Type", Value: "Workflows$XPathUserTargeting"}, {Key: "XPathConstraint", Value: value}, - } - bsonnav.DSet(actDoc, "UserTargeting", userTargeting) + }) return nil case "due_date": @@ -768,6 +766,22 @@ func (m *Mutator) Save() error { // matching on $ID. Needed when appending new keys to an activity document, // because the slice header returned by findActivityByCaption cannot propagate // appends back to the parent bson.A. +// setUserTargeting replaces a user task's targeting with targeting, in the +// current (11.2) shape and in the place of whichever key stores it: below 11.2 +// that is UserSource (ako/mxcli#1771), which the write lowers it back to +// (canon.LowerToProjectShape). Setting UserTargeting only where it already +// existed changed nothing on such a task, and the statement still reported +// success. +func (m *Mutator) setUserTargeting(actDoc bson.D, targeting bson.D) { + for i := range actDoc { + if actDoc[i].Key == "UserTargeting" || actDoc[i].Key == "UserSource" { + actDoc[i] = bson.E{Key: "UserTargeting", Value: targeting} + return + } + } + m.replaceActivity(append(actDoc, bson.E{Key: "UserTargeting", Value: targeting})) +} + func (m *Mutator) replaceActivity(updated bson.D) { actID := bsonnav.ExtractBinaryIDFromDoc(bsonnav.DGet(updated, "$ID")) if actID == "" { diff --git a/mdl/backend/wfmutator/set_targeting_1771_test.go b/mdl/backend/wfmutator/set_targeting_1771_test.go new file mode 100644 index 000000000..4b60857a9 --- /dev/null +++ b/mdl/backend/wfmutator/set_targeting_1771_test.go @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: Apache-2.0 + +package wfmutator + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" + "github.com/mendixlabs/mxcli/mdl/bsonutil" +) + +// ako/mxcli#1771: below Mendix 11.2 a stored user task holds its targeting as +// UserSource, not UserTargeting. `alter workflow … set activity '…' targeting +// …` set UserTargeting only where the key already existed, so on 9.24 and +// 10.24 it found none, changed nothing and still reported "Altered workflow" +// (measured: describe printed the old targeting afterwards). The edit replaces +// the stored targeting, whichever key holds it, in the current shape; the +// write lowers it to the project's (canon.LowerToProjectShape). +func TestWorkflowMutator_SetTargetingReplacesAStoredUserSource(t *testing.T) { + for _, tc := range []struct { + prop, value, wantType, valueKey string + }{ + {"targeting_xpath", "[Name = 'reviewer']", "Workflows$XPathUserTargeting", "XPathConstraint"}, + {"targeting_microflow", "M.ACT_Reviewers", "Workflows$MicroflowUserTargeting", "Microflow"}, + } { + for _, typ := range []string{"Workflows$UserTask", "Workflows$SingleUserTaskActivity"} { + act := makeWfActivity(typ, "Review", "task1") + act = append(act, + bson.E{Key: "UserSource", Value: bson.D{{Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, + {Key: "$Type", Value: "Workflows$XPathBasedUserSource"}, {Key: "XPathConstraint", Value: "[Name = 'admin']"}}}, + bson.E{Key: "UserTaskCompletion", Value: nil}) + sourceAt := keyIndex(act, "UserSource") + m := newMutator(makeWorkflowDoc(act)) + + if err := m.SetActivityProperty("Review", 0, tc.prop, tc.value); err != nil { + t.Fatalf("%s on a %s: %v", tc.prop, typ, err) + } + actDoc, _ := m.findActivityByCaption("Review", 0) + if bsonnav.DGet(actDoc, "UserSource") != nil { + t.Errorf("%s on a %s: the stored UserSource is still there beside the new targeting", tc.prop, typ) + } + targeting := bsonnav.DGetDoc(actDoc, "UserTargeting") + if got := bsonnav.DGetString(targeting, "$Type"); got != tc.wantType { + t.Errorf("%s on a %s: targeting %q, want %s", tc.prop, typ, got, tc.wantType) + continue + } + if got := bsonnav.DGetString(targeting, tc.valueKey); got != tc.value { + t.Errorf("%s on a %s: %s = %q, want %q", tc.prop, typ, tc.valueKey, got, tc.value) + } + // In the stored key's place, so the lowered document keeps its order. + if i := keyIndex(actDoc, "UserTargeting"); i != sourceAt { + t.Errorf("%s on a %s: targeting at %d, want the stored source's place %d", tc.prop, typ, i, sourceAt) + } + } + } +} + +// A task holding neither key gets the targeting added. +func TestWorkflowMutator_SetTargetingAddsAMissingTargeting(t *testing.T) { + m := newMutator(makeWorkflowDoc(makeWfActivity("Workflows$SingleUserTaskActivity", "Review", "task1"))) + if err := m.SetActivityProperty("Review", 0, "targeting_xpath", "[Name = 'reviewer']"); err != nil { + t.Fatal(err) + } + actDoc, _ := m.findActivityByCaption("Review", 0) + if got := bsonnav.DGetString(bsonnav.DGetDoc(actDoc, "UserTargeting"), "XPathConstraint"); got != "[Name = 'reviewer']" { + t.Errorf("targeting %q, want it added", got) + } +} + +func keyIndex(d bson.D, key string) int { + for i, e := range d { + if e.Key == key { + return i + } + } + return -1 +} diff --git a/modelsdk/mpr/writer_version_shape_1771_type_test.go b/modelsdk/mpr/writer_version_shape_1771_type_test.go index 29c747732..21a21ae5b 100644 --- a/modelsdk/mpr/writer_version_shape_1771_type_test.go +++ b/modelsdk/mpr/writer_version_shape_1771_type_test.go @@ -214,6 +214,34 @@ func TestNormalize_RefusesWhatAnOlderUserTaskCannotHold(t *testing.T) { } } +// `alter workflow … set activity '…' targeting …` on a stored Mendix 9 task +// puts a UserTargeting in the place of its UserSource (the mutator writes the +// current shape). The write lowers it back: the task keeps Studio Pro 9.24's +// key set, with the new targeting value. +func TestNormalize_EditedMendix9UserTaskTargeting(t *testing.T) { + task := bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$UserTask"}, + {Key: "Annotation", Value: nil}, {Key: "AutoAssignSingleTargetUser", Value: false}, {Key: "Caption", Value: "Edited"}, + {Key: "DueDate", Value: ""}, {Key: "Name", Value: "Edited"}, + {Key: "OnCreatedEvent", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$NoEvent"}}}, + {Key: "Outcomes", Value: bson.A{int32(3), userTaskOutcome("Done")}}, + {Key: "TaskDescription", Value: stringTemplate("")}, {Key: "TaskName", Value: stringTemplate("Edited")}, + {Key: "TaskPage", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$PageReference"}, {Key: "Page", Value: "M.TaskPage"}}}, + {Key: "UserTargeting", Value: xpathUserTargeting("[Name = 'reviewer']")}, + {Key: "UserTaskCompletion", Value: bson.D{{Key: "$ID", Value: eid()}, {Key: "$Type", Value: "Workflows$SingleInputCompletion"}}}} + d, err := normalizeShape(t, pv(9, 24, 30), workflowWithUserTasks(task)) + if err != nil { + t.Fatal(err) + } + got := findNamed(d, "Edited") + if keys := strings.Join(keysOf(got), ","); keys != studioPro924UserTask { + t.Errorf("edited task keys %s, want %s", keys, studioPro924UserTask) + } + src, _ := get(got, "UserSource") + if x, _ := get(src.(bson.D), "XPathConstraint"); x != "[Name = 'reviewer']" { + t.Errorf("edited targeting %v, want the new XPath", src) + } +} + func indexOf(d bson.D, key string) int { for i, e := range d { if e.Key == key { From 684a447f96a14b6dd010c65a0d522af852617f08 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:45:53 +0000 Subject: [PATCH 38/44] fix(executor): gate the user-task features older Mendix cannot store Symptom (ako/mxcli#1771): `mxcli check -p` passed scripts with user-task features the project's version cannot store, and exec wrote them. Group targeting on 10.24 failed mx check with CE1859, and mx convert emptied it. Once the user task is lowered to its pre-10.12 shape, a multi-user task's later rules were refused only at write time, part-way through a script. Cause: no version gate looked inside a workflow user task. The generated version data floors properties, never types, and these are type-level introductions (mendixmodelsdk 4.117.0 StructureVersionInfo): the multi-user task's UserTaskCompletion 9.22, Majority and ThresholdCompletionCriteria 10.0, MultiInputCompletion.awaitAllUsers 10.2, MicroflowCompletionCriteria 10.3, and XPath/MicroflowGroupTargeting 11.2. Fix: userTaskGates walks the activities a create workflow states, and the ones alter workflow inserts. It adds an MDL-VERSION01 gate for each clause the version lacks. A consensus or veto multi-user task with user targeting passes from 9.22. The floors are new sdk/versions entries: workflows.multi_user_task, multi_user_task_majority_threshold, multi_user_task_await_all, multi_user_task_microflow_rule and group_targeting. Measured with the bug-test (check -p, then exec, then mx check): - 9.24.30.51827: WF_Majority, WF_Await, WF_Microflow and WF_Groups reported; exec "Refusing to execute: 4 error(s) above. Nothing was written." - 10.24.24.119653: WF_Groups only. - 11.13.0: none; exec writes all five, mx check 0 errors. Control: TestCheckVersionGates_UserTaskFeatures failed for every case before the gates. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- ...9-user-task-features-gated-by-version.json | 1 + CHANGELOG.md | 1 + .../src/appendixes/version-compatibility.md | 9 +- .../1771-user-task-features-by-version.mdl | 98 ++++++++++++++++ mdl/executor/version_gates.go | 67 ++++++++++- .../version_gates_usertask_1771_test.go | 107 ++++++++++++++++++ sdk/versions/mendix-10.yaml | 16 +++ sdk/versions/mendix-11.yaml | 20 ++++ sdk/versions/mendix-9.yaml | 4 + 9 files changed, 319 insertions(+), 4 deletions(-) create mode 100644 .claude/skills/fix-issue/findings/mdl-executor/2026-10-29-user-task-features-gated-by-version.json create mode 100644 mdl-examples/bug-tests/1771-user-task-features-by-version.mdl create mode 100644 mdl/executor/version_gates_usertask_1771_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-user-task-features-gated-by-version.json b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-user-task-features-gated-by-version.json new file mode 100644 index 000000000..f65e83190 --- /dev/null +++ b/.claude/skills/fix-issue/findings/mdl-executor/2026-10-29-user-task-features-gated-by-version.json @@ -0,0 +1 @@ +{"area": "mdl/executor", "date": "2026-10-29", "symptom": "`mxcli check -p` passed scripts with user-task features the project's version cannot store, and exec wrote them: group targeting below 11.2 (CE1859 on 10.24, the targeting emptied by mx convert), and multi-user task rules and `await all users` below the version that introduced them. Once the user task was lowered to its pre-10.12 shape, these were refused only at write time, part-way through a script", "cause": "statementVersionGates had no gate for anything inside a workflow user task. The generated version data floors properties, never types, and the rule types (MajorityCompletionCriteria, ThresholdCompletionCriteria 10.0; MicroflowCompletionCriteria 10.3), the group targetings (11.2) and the multi-user task itself (UserTaskCompletion 9.22) are type-level introductions", "file": "`mdl/executor/version_gates.go` (userTaskGates, called from the CreateWorkflowStmt and AlterWorkflowStmt cases); `sdk/versions/mendix-9|10|11.yaml` (workflows.multi_user_task 9.22.0, multi_user_task_majority_threshold 10.0.0, multi_user_task_await_all 10.2.0, multi_user_task_microflow_rule 10.3.0, group_targeting 11.2.0)", "insight": "**A write-time refusal is the last line, not the check.** canon.LowerToProjectShape refuses what the older shape cannot hold, but by then the statements before it are written. Each such refusal wants a matching MDL-VERSION01 gate, so `check -p` reports it and exec refuses the script before writing anything ('Refusing to execute: 4 error(s) above. Nothing was written.' on 9.24.30). **Gate the clause, not the statement**: a workflow with a consensus or veto multi-user task is fine from 9.22, so the gates walk the activities (create, and every activity alter workflow inserts) and fire only for a rule, a targeting kind or `await all users` the version lacks. **Read the type floors from mendixmodelsdk's StructureVersionInfo** (gen/workflows.js; text-extract it, never execute it). The generated version.go has no row for a type. Measured with the bug-test: check -p on 9.24.30 reports WF_Majority (10.0), WF_Await (10.2), WF_Microflow (10.3) and WF_Groups (11.2). 10.24.24 reports WF_Groups only. 11.13.0 reports none, exec writes all five, and mx check gives 0 errors. Control: TestCheckVersionGates_UserTaskFeatures failed for every case before the gates. Tests: TestCheckVersionGates_UserTaskFeatures, …AtTheirFloors; mdl-examples/bug-tests/1771-user-task-features-by-version.mdl", "refs": ["#1771"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b5f0ce2d..265ecc5a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **User-task features an older Mendix cannot store are refused up front** (ako/mxcli#1771) — `mxcli check -p` and `exec` report MDL-VERSION01 before anything is written, naming the clause. The floors are group targeting (`targeting groups …`) 11.2, a multi-user task 9.22, `decide by majority` or `threshold` 10.0, `await all users` 10.2, and `decide by microflow` 10.3. Before, group targeting was written on 10.24 and failed `mx check` with CE1859. The others were refused only at write time, after the statements before them were written. Measured on 9.24.30 and 10.24.24 (each refused, nothing written) and on 11.13.0 (all written, 0 errors). - **`alter workflow … set (Targeting: …)` changes a user task's targeting on Mendix 9 to 11.1** (ako/mxcli#1771) — it reported "Altered workflow" and changed nothing. Before 11.2 a user task stores its targeting under another key, and the edit only replaced a key that was already there. The edit now replaces the stored targeting, whichever key holds it, and the write stores it in the version's shape. Measured on 9.24.30 and 10.24.24: `describe` shows the new targeting, with 0 errors. 11.2 and later are unchanged. - **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. - **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index f5c8c1325..83fc73373 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -119,6 +119,11 @@ body on Mendix 10 too, and Studio Pro could not open the project. |---------|-----------|-----|-------| | Basic workflows | `CREATE WORKFLOW` | Yes | Yes | | User tasks | `USER TASK ... OUTCOMES (...)` | Yes | Yes | +| Multi-user tasks | `MULTI USER TASK ...` | 9.22+ | Yes | +| Multi-user task rules majority and threshold | `DECIDE BY MAJORITY ...`, `DECIDE BY THRESHOLD ...` | No | Yes | +| Waiting for all users | `AWAIT ALL USERS` | No | 10.2+ | +| Deciding by a microflow | `DECIDE BY MICROFLOW ...` | No | 10.3+ | +| Group targeting | `TARGETING GROUPS XPATH ...`, `TARGETING GROUPS MICROFLOW ...` | No | 11.2+ | | Parallel splits | `PARALLEL SPLIT` | Yes | Yes | Below Mendix 11.3 a workflow's main flow has no stored start or end activity: @@ -199,8 +204,8 @@ What an older shape cannot hold is refused, naming the element, rather than drop - a data view fed by another widget below 10.0; - a private constant value in a configuration below 10.9; - a page's flow call argument that is any other expression (a literal, an attribute path) below 10.10; -- a user task targeting groups (`targeting groups xpath …`, `targeting groups microflow …`) below 11.2, where a task targets users only ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); -- a multi-user task below 9.22, `decide by majority` or `decide by threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14 ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); +- a user task targeting groups (`targeting groups xpath …`, `targeting groups microflow …`) below 11.2, where a task targets users only; +- a multi-user task below 9.22, `decide by majority` or `decide by threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. The user-task features are also refused up front, by `check -p` and by `exec` before anything is written, with MDL-VERSION01 ([ako/mxcli#1771](https://github.com/ako/mxcli/issues/1771)); - an aggregate over an expression (`sum $L of …`), `reduce`, `all` or `any` below 10.0. The plain `count`, `sum`, `average`, `minimum` and `maximum` are written on every version ([ako/mxcli#1163](https://github.com/ako/mxcli/issues/1163)). Before this, `describe` on 9.24 printed none of these properties, so a describe → exec round trip removed them. Writing a data view on 9.24 was refused outright. diff --git a/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl b/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl new file mode 100644 index 000000000..f84d8326f --- /dev/null +++ b/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl @@ -0,0 +1,98 @@ +mdl 1; +-- ako/mxcli#1771: what an older user task cannot store is refused up front +-- with MDL-VERSION01, by `mxcli check -p` and by exec before anything is +-- written. Before, these were written as they are on 11.2+: +-- - group targeting below 11.2. mx check 10.24.24 then reported CE1859 +-- "XPath constraint is required", and `mx convert` emptied the +-- targeting. Before 11.2 a user task targets users only. +-- - a multi-user task below 10.12, with the 10.12 types. mx check 9.24.30 +-- could not load the project (TypeCacheUnknownTypeException). +-- Below 10.12 a multi-user task is now written as a Workflows$UserTask (see +-- 1771-user-task-below-10-12.mdl), which has no room for what came later: +-- multi user task 9.22 +-- decide by majority / threshold 10.0 +-- await all users 10.2 +-- decide by microflow 10.3 +-- targeting groups xpath / microflow 11.2 +-- (mendixmodelsdk 4.117.0 StructureVersionInfo.) +-- +-- EXPECTED: `mxcli check` passes this file. With -p on a project below a +-- floor, each workflow that states the feature is reported as MDL-VERSION01, +-- naming it. Measured on fresh apps: +-- 9.24.30.51827 WF_Majority, WF_Await, WF_Microflow, WF_Groups +-- 10.24.24.119653 WF_Groups +-- 11.13.0 none; exec writes all five, and mx check reports 0 errors. + +create module Probe1771Gt; + +create persistent entity Probe1771Gt.Request ( Title: String(100) ); + +create microflow Probe1771Gt.ACT_Groups ($Workflow: System.Workflow, $Context: Probe1771Gt.Request) +returns list of System.WorkflowGroup as $Groups +begin + retrieve $Groups from System.WorkflowGroup; + return $Groups; +end; + +create microflow Probe1771Gt.ACT_Decide ($WorkflowUserTask: System.WorkflowUserTask) +returns String as $Outcome +begin + return 'Yes'; +end; + +create page Probe1771Gt.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txt1 (content: 'Task') +}; + +create workflow Probe1771Gt.WF_Veto + parameter $WorkflowContext: Probe1771Gt.Request +begin + multi user task Vote 'Vote' + page Probe1771Gt.TaskPage + targeting users xpath [Name = 'admin'] + decide by veto 'No' + outcomes 'Yes' { } 'No' { }; +end workflow; + +create workflow Probe1771Gt.WF_Majority + parameter $WorkflowContext: Probe1771Gt.Request +begin + multi user task Vote 'Vote' + page Probe1771Gt.TaskPage + targeting users xpath [Name = 'admin'] + decide by majority more than half fallback 'No' + outcomes 'Yes' { } 'No' { }; +end workflow; + +create workflow Probe1771Gt.WF_Await + parameter $WorkflowContext: Probe1771Gt.Request +begin + multi user task Vote 'Vote' + page Probe1771Gt.TaskPage + targeting users xpath [Name = 'admin'] + await all users + outcomes 'Yes' { } 'No' { }; +end workflow; + +create workflow Probe1771Gt.WF_Microflow + parameter $WorkflowContext: Probe1771Gt.Request +begin + multi user task Vote 'Vote' + page Probe1771Gt.TaskPage + targeting users xpath [Name = 'admin'] + decide by microflow Probe1771Gt.ACT_Decide + outcomes 'Yes' { } 'No' { }; +end workflow; + +create workflow Probe1771Gt.WF_Groups + parameter $WorkflowContext: Probe1771Gt.Request +begin + user task Review 'Review' + page Probe1771Gt.TaskPage + targeting groups microflow Probe1771Gt.ACT_Groups + outcomes 'Done' { }; +end workflow; diff --git a/mdl/executor/version_gates.go b/mdl/executor/version_gates.go index 33514acad..e2afaec8e 100644 --- a/mdl/executor/version_gates.go +++ b/mdl/executor/version_gates.go @@ -201,7 +201,7 @@ func statementVersionGates(stmt ast.Statement) []versionGate { if workflowUsesNotificationEvents(activities) { gates = append(gates, notificationEventsGate) } - return gates + return append(gates, userTaskGates(activities)...) case *ast.AlterWorkflowStmt: // Workflows require Mendix 9.12+. gates := []versionGate{{"workflows", "basic", "alter workflow", @@ -213,11 +213,74 @@ func statementVersionGates(stmt ast.Statement) []versionGate { if workflowUsesNotificationEvents(added) { gates = append(gates, notificationEventsGate) } - return gates + return append(gates, userTaskGates(added)...) } return nil } +// userTaskGates are the gates of what the user tasks among activities state +// that an older user task cannot store, each once (ako/mxcli#1771). Below 11.2 +// a task targets users only, and a multi-user task, its later rules and +// `await all users` arrived in 9.22, 10.0, 10.3 and 10.2 (mendixmodelsdk +// 4.117.0). The write refuses them too (canon.LowerToProjectShape); this names +// the statement before anything is written. +func userTaskGates(activities []ast.WorkflowActivityNode) []versionGate { + var groups, multi, majority, threshold, microflow, await bool + walkWorkflowActivities(activities, func(a ast.WorkflowActivityNode) { + ut, ok := a.(*ast.WorkflowUserTaskNode) + if !ok { + return + } + switch ut.Targeting.Kind { + case "group_xpath", "group_microflow": + groups = true + } + if !ut.IsMultiUser { + return + } + multi = true + await = await || ut.AwaitAllUsers + if ut.Completion != nil { + switch ut.Completion.Rule { + case "majority": + majority = true + case "threshold": + threshold = true + case "microflow": + microflow = true + } + } + }) + var gates []versionGate + if groups { + gates = append(gates, versionGate{"workflows", "group_targeting", "targeting groups on a user task", + "Before Mendix 11.2 a user task targets users only: `targeting users xpath […]`, or a microflow " + + "returning users. Workflow groups arrived in 11.2."}) + } + if multi { + gates = append(gates, versionGate{"workflows", "multi_user_task", "multi user task", + "Mendix added multi-user tasks in 9.22: use a user task, or upgrade the project"}) + } + for _, r := range []struct { + used bool + rule string + }{{majority, "majority"}, {threshold, "threshold"}} { + if r.used { + gates = append(gates, versionGate{"workflows", "multi_user_task_majority_threshold", "decide by " + r.rule, + "Mendix added the majority and threshold rules in 10.0: decide by consensus or veto, or upgrade the project"}) + } + } + if microflow { + gates = append(gates, versionGate{"workflows", "multi_user_task_microflow_rule", "decide by microflow", + "Mendix added deciding a multi-user task by a microflow in 10.3: decide by consensus or veto, or upgrade the project"}) + } + if await { + gates = append(gates, versionGate{"workflows", "multi_user_task_await_all", "await all users", + "Mendix added `await all users` in 10.2: leave it out, so the task completes once its rule is met, or upgrade the project"}) + } + return gates +} + // The floors of a flow call's `returns ( … )` mappings (ako/mxcli#977). The // builder applies the same two, for the widgets this walk cannot see (a // fragment's, expanded at build time). diff --git a/mdl/executor/version_gates_usertask_1771_test.go b/mdl/executor/version_gates_usertask_1771_test.go new file mode 100644 index 000000000..1d11cf45d --- /dev/null +++ b/mdl/executor/version_gates_usertask_1771_test.go @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// ako/mxcli#1771: what an older user task cannot store is refused by +// `check -p` and by exec's pre-flight with MDL-VERSION01, before anything is +// written. The write-time refusal in canon.LowerToProjectShape stays as the +// last line; these name the statement and say what to do instead. +// - Group targeting is 11.2. Before it, a user task targets users only. +// - A multi-user task is 9.22. +// - Its rules majority and threshold are 10.0, microflow is 10.3, and +// `await all users` is 10.2 (mendixmodelsdk 4.117.0). +func TestCheckVersionGates_UserTaskFeatures(t *testing.T) { + task := func(f func(*ast.WorkflowUserTaskNode)) *ast.WorkflowUserTaskNode { + n := &ast.WorkflowUserTaskNode{Name: "Review", Caption: "Review"} + f(n) + return n + } + groups := task(func(n *ast.WorkflowUserTaskNode) { + n.Targeting = ast.WorkflowTargetingNode{Kind: "group_xpath", XPath: "[Name = 'Reviewers']"} + }) + groupsMicroflow := task(func(n *ast.WorkflowUserTaskNode) { + n.Targeting = ast.WorkflowTargetingNode{Kind: "group_microflow", Microflow: ast.QualifiedName{Module: "M", Name: "Groups"}} + }) + multi := task(func(n *ast.WorkflowUserTaskNode) { n.IsMultiUser = true }) + rule := func(r string) *ast.WorkflowUserTaskNode { + return task(func(n *ast.WorkflowUserTaskNode) { + n.IsMultiUser = true + n.Completion = &ast.WorkflowCompletionRuleNode{Rule: r} + }) + } + await := task(func(n *ast.WorkflowUserTaskNode) { n.IsMultiUser, n.AwaitAllUsers = true, true }) + + for _, c := range []struct { + name string + act ast.WorkflowActivityNode + below [3]int + want string + }{ + {"groups xpath", groups, [3]int{11, 1, 0}, "targeting groups"}, + {"groups microflow", groupsMicroflow, [3]int{10, 24, 24}, "targeting groups"}, + {"multi user task", multi, [3]int{9, 21, 0}, "multi user task"}, + {"majority", rule("majority"), [3]int{9, 24, 30}, "decide by majority"}, + {"threshold", rule("threshold"), [3]int{9, 24, 30}, "decide by threshold"}, + {"microflow rule", rule("microflow"), [3]int{10, 2, 0}, "decide by microflow"}, + {"await all users", await, [3]int{10, 1, 0}, "await all users"}, + } { + for _, stmt := range []ast.Statement{ + &ast.CreateWorkflowStmt{Activities: []ast.WorkflowActivityNode{c.act}}, + &ast.AlterWorkflowStmt{Operations: []ast.AlterWorkflowOp{ + &ast.InsertAfterOp{NewActivities: []ast.WorkflowActivityNode{c.act}}}}, + } { + prog := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, stmt}} + vs := versionGateViolations(versionCtx(t, c.below[0], c.below[1], c.below[2]), prog) + if len(vs) != 1 || vs[0].RuleID != "MDL-VERSION01" || !strings.Contains(vs[0].Message, c.want) || vs[0].Suggestion == "" { + t.Errorf("%s (%T) below its floor: want one MDL-VERSION01 naming %q with a hint, got %+v", c.name, stmt, c.want, vs) + } + // Control: a version with every feature passes. + if vs := versionGateViolations(versionCtx(t, 11, 99, 0), prog); len(vs) != 0 { + t.Errorf("%s (%T): reported at 11.99: %+v", c.name, stmt, vs) + } + } + } +} + +// The floors are each feature's own: a user task targeting users, and a +// multi-user task deciding by consensus or veto, are written from 9.22. +func TestCheckVersionGates_UserTaskFeaturesAtTheirFloors(t *testing.T) { + prog := &ast.Program{Statements: []ast.Statement{ + &ast.CreateModuleStmt{Name: "M"}, + &ast.CreateWorkflowStmt{Activities: []ast.WorkflowActivityNode{ + &ast.WorkflowUserTaskNode{Name: "A", Targeting: ast.WorkflowTargetingNode{Kind: "xpath", XPath: "[Name = 'a']"}}, + &ast.WorkflowUserTaskNode{Name: "B", Targeting: ast.WorkflowTargetingNode{Kind: "microflow", Microflow: ast.QualifiedName{Module: "M", Name: "U"}}}, + &ast.WorkflowUserTaskNode{Name: "C", IsMultiUser: true, Completion: &ast.WorkflowCompletionRuleNode{Rule: "veto", Veto: "No"}}, + &ast.WorkflowUserTaskNode{Name: "D", IsMultiUser: true, Completion: &ast.WorkflowCompletionRuleNode{Rule: "consensus"}}, + }}, + }} + if vs := versionGateViolations(versionCtx(t, 9, 22, 0), prog); len(vs) != 0 { + t.Errorf("user targeting and consensus/veto multi-user tasks reported on 9.22: %+v", vs) + } + for _, v := range [][3]int{{10, 0, 0}, {10, 2, 0}, {10, 3, 0}, {11, 2, 0}} { + for _, act := range []*ast.WorkflowUserTaskNode{ + {Name: "M", IsMultiUser: true, Completion: &ast.WorkflowCompletionRuleNode{Rule: "majority"}}, + {Name: "G", Targeting: ast.WorkflowTargetingNode{Kind: "group_xpath", XPath: "[Name = 'g']"}}, + } { + floor := [3]int{10, 0, 0} + if act.Name == "G" { + floor = [3]int{11, 2, 0} + } + if v[0] < floor[0] || (v[0] == floor[0] && v[1] < floor[1]) { + continue + } + p := &ast.Program{Statements: []ast.Statement{&ast.CreateModuleStmt{Name: "M"}, + &ast.CreateWorkflowStmt{Activities: []ast.WorkflowActivityNode{act}}}} + if vs := versionGateViolations(versionCtx(t, v[0], v[1], v[2]), p); len(vs) != 0 { + t.Errorf("%s reported on %v, at or past its floor: %+v", act.Name, v, vs) + } + } + } +} diff --git a/sdk/versions/mendix-10.yaml b/sdk/versions/mendix-10.yaml index 207443cab..96a25823d 100644 --- a/sdk/versions/mendix-10.yaml +++ b/sdk/versions/mendix-10.yaml @@ -211,6 +211,22 @@ features: min_version: "9.0.0" user_task: min_version: "9.0.0" + multi_user_task: + min_version: "9.22.0" + mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" + notes: "Before 10.12 a Workflows$UserTask whose UserTaskCompletion is a MultiInputCompletion, introduced with consensus and veto in 9.22.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_majority_threshold: + min_version: "10.0.0" + mdl: "MULTI USER TASK … DECIDE BY MAJORITY MORE THAN HALF | DECIDE BY THRESHOLD 60 PERCENT" + notes: "MajorityCompletionCriteria and ThresholdCompletionCriteria introduced in 10.0.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_await_all: + min_version: "10.2.0" + mdl: "MULTI USER TASK … AWAIT ALL USERS" + notes: "MultiInputCompletion.awaitAllUsers introduced in 10.2.0, on MultiUserTaskActivity from 10.12 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_microflow_rule: + min_version: "10.3.0" + mdl: "MULTI USER TASK … DECIDE BY MICROFLOW M.Decide" + notes: "MicroflowCompletionCriteria introduced in 10.3.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." event_handlers: min_version: "10.7.0" mdl: "CREATE WORKFLOW … ON WORKFLOW EVENTS (…) MICROFLOW Module.Handler AS '…'" diff --git a/sdk/versions/mendix-11.yaml b/sdk/versions/mendix-11.yaml index 02a08812e..ef0935f85 100644 --- a/sdk/versions/mendix-11.yaml +++ b/sdk/versions/mendix-11.yaml @@ -244,6 +244,26 @@ features: min_version: "9.0.0" user_task: min_version: "9.0.0" + multi_user_task: + min_version: "9.22.0" + mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" + notes: "Before 10.12 a Workflows$UserTask whose UserTaskCompletion is a MultiInputCompletion, introduced with consensus and veto in 9.22.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_majority_threshold: + min_version: "10.0.0" + mdl: "MULTI USER TASK … DECIDE BY MAJORITY MORE THAN HALF | DECIDE BY THRESHOLD 60 PERCENT" + notes: "MajorityCompletionCriteria and ThresholdCompletionCriteria introduced in 10.0.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_await_all: + min_version: "10.2.0" + mdl: "MULTI USER TASK … AWAIT ALL USERS" + notes: "MultiInputCompletion.awaitAllUsers introduced in 10.2.0, on MultiUserTaskActivity from 10.12 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + multi_user_task_microflow_rule: + min_version: "10.3.0" + mdl: "MULTI USER TASK … DECIDE BY MICROFLOW M.Decide" + notes: "MicroflowCompletionCriteria introduced in 10.3.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." + group_targeting: + min_version: "11.2.0" + mdl: "USER TASK … TARGETING GROUPS XPATH [Name = 'Reviewers'] | TARGETING GROUPS MICROFLOW M.Groups" + notes: "XPathGroupTargeting and MicroflowGroupTargeting introduced in 11.2.0 with UserTaskActivity.userTargeting (mendixmodelsdk 4.117.0). Before 11.2 a user task's UserSource targets users only (ako/mxcli#1771)." ai_agent_task: min_version: "11.9.0" mdl: "CALL AGENT MICROFLOW Module.InvokeAgent WITH (Param = '$WorkflowContext')" diff --git a/sdk/versions/mendix-9.yaml b/sdk/versions/mendix-9.yaml index 7995a0554..433c27531 100644 --- a/sdk/versions/mendix-9.yaml +++ b/sdk/versions/mendix-9.yaml @@ -61,6 +61,10 @@ features: min_version: "9.0.0" user_task: min_version: "9.0.0" + multi_user_task: + min_version: "9.22.0" + mdl: "MULTI USER TASK Vote 'Vote' PAGE M.TaskPage OUTCOMES 'Yes' { } 'No' { };" + notes: "Before 10.12 a Workflows$UserTask whose UserTaskCompletion is a MultiInputCompletion, introduced with consensus and veto in 9.22.0 (mendixmodelsdk 4.117.0; ako/mxcli#1771)." navigation: profiles: From b47de7efcbd48e9c5d2fd4c34d93c194949096b7 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 07:53:34 +0000 Subject: [PATCH 39/44] test(doctype): write user tasks in the Mendix 10 workflow script 24-workflow-examples.mdl, the only doctype script with user tasks, is gated 11.0+ as a whole. CI's 10.24 leg therefore never wrote a user task. That hid ako/mxcli#1771: every user task mxcli wrote on 10.12 to 11.1 failed mx check with CE1859. 24b-workflow-mendix10-examples.mdl (10.0+) now writes user tasks by XPath, by a microflow and with no targeting. It also writes multi-user tasks with the consensus, veto, majority and threshold rules and participants, which 10.0 has. Then it inserts a task with alter workflow and changes a task's targeting. Measured with TestMxCheck_DoctypeScripts/24b: - 10.24.24.119653: PASS, mx check 0 errors. Control: with the targeting lowering stubbed, it FAILS with CE1859 on all 8 user tasks. - 11.13.0: PASS (24, 24b and 25 all pass, on both versions). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .../24b-workflow-mendix10-examples.mdl | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/mdl-examples/doctype-tests/24b-workflow-mendix10-examples.mdl b/mdl-examples/doctype-tests/24b-workflow-mendix10-examples.mdl index 8d504c304..665a85a5a 100644 --- a/mdl-examples/doctype-tests/24b-workflow-mendix10-examples.mdl +++ b/mdl-examples/doctype-tests/24b-workflow-mendix10-examples.mdl @@ -19,6 +19,13 @@ mdl 1; -- `annotation` is the workflow's own note; from 11.3 they are stored and the -- note is the start activity's. A branch's `end workflow` is stored on every -- version. +-- +-- User tasks (end of file) hid ako/mxcli#1771 the same way. Below 11.2 a task's +-- targeting is a UserSource, and every user task mxcli wrote on 10.24 failed mx +-- check with CE1859 "XPath constraint is required". Only the features 10.0 +-- has are used: user targeting, and the consensus, veto, majority and +-- threshold rules. Group targeting is 11.2, `await all users` 10.2 and +-- `decide by microflow` 10.3. -- ============================================================================ create module WfTen; @@ -75,3 +82,68 @@ begin false -> { end workflow; } ; end workflow; + +-- User tasks: users by XPath, users by a microflow, no targeting, and +-- multi-user tasks with each rule 10.0 has. +create microflow WfTen.ACT_Approvers ($Workflow: System.Workflow, $Context: WfTen.Request) +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +create page WfTen.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: ( $WorkflowUserTask: System.WorkflowUserTask ) +) { + dynamictext txtTask (content: 'Task') +}; + +create workflow WfTen.WF_Review + parameter $WorkflowContext: WfTen.Request + display 'Review' +begin + user task Review 'Review' + page WfTen.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'Approve' { } 'Reject' { end workflow; }; + user task Handle 'Handle' + page WfTen.TaskPage + targeting users microflow WfTen.ACT_Approvers + outcomes 'Done' { }; + user task Anyone 'Anyone' + page WfTen.TaskPage + outcomes 'Done' { }; + multi user task Agree 'Agree' + page WfTen.TaskPage + targeting users microflow WfTen.ACT_Approvers + outcomes 'Yes' { } 'No' { }; + multi user task Vote 'Vote' + page WfTen.TaskPage + targeting users xpath [Name = 'admin'] + participants 2 + decide by veto 'No' + outcomes 'Yes' { } 'No' { }; + multi user task Poll 'Poll' + page WfTen.TaskPage + targeting users microflow WfTen.ACT_Approvers + participants 50 percent + decide by majority more than half fallback 'No' + outcomes 'Yes' { } 'No' { }; + multi user task Quorum 'Quorum' + page WfTen.TaskPage + targeting users microflow WfTen.ACT_Approvers + decide by threshold 60 percent fallback 'No' + outcomes 'Yes' { } 'No' { }; +end workflow; + +-- ALTER inserts a user task through the mutator, and changes a task's +-- targeting in whichever key the version stores it. +alter workflow WfTen.WF_Review { + insert after Handle { user task Confirm 'Confirm' + page WfTen.TaskPage + targeting users xpath [Name = 'admin'] + outcomes 'OK' { }; } +}; +alter workflow WfTen.WF_Review { set (Targeting: xpath [Name = 'reviewer']) on Review; }; From ee1a059196bd19ef868f01b3db3ac0ebaa573844 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 08:29:47 +0000 Subject: [PATCH 40/44] test(bug-tests): measure the 1771 user-task shapes on Mendix 10.11 The 10.0-10.11 claims rested on mendixmodelsdk's version data and on the 9.24 measurements. They are now measured on 10.11.0.37074, the last version before the 10.12 type boundary (mxbuild downloaded for this, then deleted): - With the user-task lowering stubbed, a user task makes the project unloadable (TypeCacheUnknownTypeException ... Workflows$SingleUserTaskActivity). - With the fix, the 1771 bug-tests give 0 errors. The multi-user tasks' MultiInputCompletion holds AwaitAllUsers (10.2+). mx convert 10.11 changes nothing, and describe -> exec reports "Unchanged workflow". Majority, await all users and a microflow rule check clean, and alter workflow's targeting edit lands. 1771-user-task-features-by-version.mdl's group-targeted task used a microflow returning System.WorkflowGroup, an entity Mendix introduced in 11.2. Written on 10.11 it was CE1613, unrelated to the user task. The task now targets groups by XPath. Re-measured: check -p reports the expected gates on 9.24.30, 10.11.0 and 10.24.24. On 11.13.0 exec writes all five workflows with 0 errors. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82sUVQEB3K5W5vh85wM --- .../2026-10-29-user-task-type-below-10-12.json | 2 +- CHANGELOG.md | 2 +- .../1771-alter-user-task-targeting.mdl | 1 + .../bug-tests/1771-user-task-below-10-12.mdl | 4 ++++ .../1771-user-task-features-by-version.mdl | 17 +++++++---------- .../1771-user-task-targeting-below-11-2.mdl | 2 ++ 6 files changed, 16 insertions(+), 12 deletions(-) diff --git a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json index 6421029ff..137796213 100644 --- a/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json +++ b/.claude/skills/fix-issue/findings/modelsdk/2026-10-29-user-task-type-below-10-12.json @@ -1 +1 @@ -{"area": "modelsdk", "date": "2026-10-29", "symptom": "On Mendix 9.x a workflow with a user task made the project unloadable. exec reported 'Created workflow', then mx check 9.24.30 and Studio Pro died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity`. The same happened on 10.0–10.11 by the SDK's version data, which was not measured there", "cause": "userTaskToGen (and the ALTER mutator's inserts, through it) writes Workflows$SingleUserTaskActivity / MultiUserTaskActivity, TYPES introduced in 10.12.0, which also deleted Workflows$UserTask (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared the written $Type with the project's version. Before 10.12 the kind of task is the UserTask's UserTaskCompletion (9.22): SingleInputCompletion, or MultiInputCompletion {CompletionCriteria, TargetUserInput, AwaitAllUsers from 10.2}", "file": "`modelsdk/canon/versionshape_usertask.go` (lowerUserTaskType, dropTaskBoundaryEvents, userTaskActivitySince, completionCriteriaSince), `mdl/backend/modelsdk/workflow_read.go` (UserTask case reads the MultiInputCompletion), `mdl/executor/validate_workflow_rewrite.go` (rawMultiUserTaskSettings sees a 9.x multi-user task)", "insight": "**A type rename can also be a restructure.** Converting MultiUserTaskActivity down is more than a $Type swap. Three of its keys move into a nested MultiInputCompletion, and the outcomes stay on the task, so the CompletionCriteria's outcome pointers still resolve. Studio Pro 9.24.3's key set is the target: Multilevel Dynamic Workflow (Marketplace 221741) stores `$ID,$Type,Annotation,AutoAssignSingleTargetUser,Caption,DueDate,Name,OnCreatedEvent,Outcomes,TaskDescription,TaskName,TaskPage,UserSource,UserTaskCompletion`. **With no Studio Pro sample of a 9.x multi-user task, convert UP as the oracle.** The 9.24 output, after `mx convert -p` 10.24.24, has user tasks identical in keys and values to what mxcli writes on 10.24 directly, rule and participants included. **Every reader of the old shape needs the restructure too.** describe read a 9.x multi-user task as a single one, and the create-or-modify guard looked for the rule only on MultiUserTaskActivity, so a 9.24 rewrite would reset the rule silently. **Give an element the lowering ADDS a derived $ID** (completionID from the task's $ID), so the same task lowers to the same bytes. The boundary events on a user task (10.14, declared on the abstract UserTaskActivity) slipped past the floor the same way: an empty list is dropped below 10.14, and a boundary event is refused. Measured: bug-test on fresh apps. 9.24.30 before TypeCacheUnknownTypeException, after 0 errors. `mx convert -p` 9.24.30 changes no key and no value. describe → exec 'Unchanged workflow'. 10.24.24 and 11.13.0 give 0 errors. The CE2513 'Java 11' after converting up is in an untouched 9.24 app too. Control: stubbing lowerUserTaskType fails TestNormalize_UserTaskIsAWorkflowsUserTaskBelowMendix10_12, …MultiUserTask… and …RefusesWhatAnOlderUserTaskCannotHold with the 10.12 types on 9.24. The reader and guard tests failed before their fixes. Tests: those three, TestNormalize_UserTaskCompletionIDIsStable, TestWorkflowRead_UserTaskInTheMendix9Shape, TestWorkflowRewrite_SeesAMendix9MultiUserTasksSettings; mdl-examples/bug-tests/1771-user-task-below-10-12.mdl", "refs": ["#1771", "#1163"]} +{"area": "modelsdk", "date": "2026-10-29", "symptom": "On Mendix 9.x a workflow with a user task made the project unloadable. exec reported 'Created workflow', then mx check 9.24.30 and Studio Pro died with `TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity`. The same happened on 10.11.0 (measured), and on all of 10.0–10.11 by the SDK's version data", "cause": "userTaskToGen (and the ALTER mutator's inserts, through it) writes Workflows$SingleUserTaskActivity / MultiUserTaskActivity, TYPES introduced in 10.12.0, which also deleted Workflows$UserTask (mendixmodelsdk 4.117.0). The generated version data holds property floors only, so nothing compared the written $Type with the project's version. Before 10.12 the kind of task is the UserTask's UserTaskCompletion (9.22): SingleInputCompletion, or MultiInputCompletion {CompletionCriteria, TargetUserInput, AwaitAllUsers from 10.2}", "file": "`modelsdk/canon/versionshape_usertask.go` (lowerUserTaskType, dropTaskBoundaryEvents, userTaskActivitySince, completionCriteriaSince), `mdl/backend/modelsdk/workflow_read.go` (UserTask case reads the MultiInputCompletion), `mdl/executor/validate_workflow_rewrite.go` (rawMultiUserTaskSettings sees a 9.x multi-user task)", "insight": "**A type rename can also be a restructure.** Converting MultiUserTaskActivity down is more than a $Type swap. Three of its keys move into a nested MultiInputCompletion, and the outcomes stay on the task, so the CompletionCriteria's outcome pointers still resolve. Studio Pro 9.24.3's key set is the target: Multilevel Dynamic Workflow (Marketplace 221741) stores `$ID,$Type,Annotation,AutoAssignSingleTargetUser,Caption,DueDate,Name,OnCreatedEvent,Outcomes,TaskDescription,TaskName,TaskPage,UserSource,UserTaskCompletion`. **With no Studio Pro sample of a 9.x multi-user task, convert UP as the oracle.** The 9.24 output, after `mx convert -p` 10.24.24, has user tasks identical in keys and values to what mxcli writes on 10.24 directly, rule and participants included. **Every reader of the old shape needs the restructure too.** describe read a 9.x multi-user task as a single one, and the create-or-modify guard looked for the rule only on MultiUserTaskActivity, so a 9.24 rewrite would reset the rule silently. **Give an element the lowering ADDS a derived $ID** (completionID from the task's $ID), so the same task lowers to the same bytes. The boundary events on a user task (10.14, declared on the abstract UserTaskActivity) slipped past the floor the same way: an empty list is dropped below 10.14, and a boundary event is refused. Measured: bug-test on fresh apps. 9.24.30 before TypeCacheUnknownTypeException, after 0 errors. `mx convert -p` 9.24.30 changes no key and no value. describe → exec 'Unchanged workflow'. 10.24.24 and 11.13.0 give 0 errors. The CE2513 'Java 11' after converting up is in an untouched 9.24 app too. 10.11.0.37074 (downloaded for this, then deleted): before (lowering stubbed) TypeCacheUnknownTypeException, which confirms the SDK's 10.12 type boundary on the platform. After, 0 errors with AwaitAllUsers in the MultiInputCompletion, mx convert 10.11 changes nothing, and describe → exec 'Unchanged workflow'. Majority, await all users and a microflow rule written on 10.11 check clean. Control: stubbing lowerUserTaskType fails TestNormalize_UserTaskIsAWorkflowsUserTaskBelowMendix10_12, …MultiUserTask… and …RefusesWhatAnOlderUserTaskCannotHold with the 10.12 types on 9.24. The reader and guard tests failed before their fixes. Tests: those three, TestNormalize_UserTaskCompletionIDIsStable, TestWorkflowRead_UserTaskInTheMendix9Shape, TestWorkflowRewrite_SeesAMendix9MultiUserTasksSettings; mdl-examples/bug-tests/1771-user-task-below-10-12.mdl", "refs": ["#1771", "#1163"]} diff --git a/CHANGELOG.md b/CHANGELOG.md index 265ecc5a4..1084df4b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **User-task features an older Mendix cannot store are refused up front** (ako/mxcli#1771) — `mxcli check -p` and `exec` report MDL-VERSION01 before anything is written, naming the clause. The floors are group targeting (`targeting groups …`) 11.2, a multi-user task 9.22, `decide by majority` or `threshold` 10.0, `await all users` 10.2, and `decide by microflow` 10.3. Before, group targeting was written on 10.24 and failed `mx check` with CE1859. The others were refused only at write time, after the statements before them were written. Measured on 9.24.30 and 10.24.24 (each refused, nothing written) and on 11.13.0 (all written, 0 errors). - **`alter workflow … set (Targeting: …)` changes a user task's targeting on Mendix 9 to 11.1** (ako/mxcli#1771) — it reported "Altered workflow" and changed nothing. Before 11.2 a user task stores its targeting under another key, and the edit only replaced a key that was already there. The edit now replaces the stored targeting, whichever key holds it, and the write stores it in the version's shape. Measured on 9.24.30 and 10.24.24: `describe` shows the new targeting, with 0 errors. 11.2 and later are unchanged. -- **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. +- **A workflow with a user task no longer makes a Mendix 9 or 10.0–10.11 project unloadable** (ako/mxcli#1771) — `exec` reported success, then `mx check` and Studio Pro could not load the project at all: "TypeCacheUnknownTypeException: The type cache does not contain a type with qualified name Workflows$SingleUserTaskActivity". mxcli wrote a user task as a `SingleUserTaskActivity` or `MultiUserTaskActivity`, types introduced in 10.12. Before 10.12 both are a `Workflows$UserTask`, as Studio Pro 9.24 stores it. Its completion says which kind it is, and holds a multi-user task's rule, participants and, from 10.2, `await all users`. That is what is written now, for `create`, `create or modify` and a task `alter workflow` inserts. `describe` reads a multi-user task in that shape, and `create or modify` no longer resets its rule silently. What the older type cannot hold is refused, naming the task: a multi-user task below 9.22, `decide by majority` or `threshold` below 10.0, `decide by microflow` below 10.3, `await all users` below 10.2, and a boundary event on a user task below 10.14. Measured on 9.24.30 and 10.11.0: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. Converted up to 10.24, the tasks are the ones mxcli writes there directly. 10.12 and later are unchanged. - **A user task keeps its targeting on Mendix 10.12 to 11.1, including 10.24** (ako/mxcli#1771) — every user task mxcli wrote there failed `mx check` with CE1859 "XPath constraint is required", including tasks targeted by a microflow and multi-user tasks. `exec` and `mxcli check` were clean. mxcli stored the targeting as `UserTargeting`, which Mendix introduced in 11.2. The older versions read the task as having no targeting at all, and `mx convert` replaced it with an empty XPath constraint. Before 11.2 the same fact is a `UserSource` holding the same value, measured by converting Studio Pro-authored 9.x workflows up to 10.24 and 11.13. That is what is written now, for users by XPath, users by a microflow, and no targeting. Group targeting is 11.2 and is refused below it, naming the task. Measured on 10.24.24: 0 errors, `mx convert` changes nothing, and `describe` → `exec` reports the workflow unchanged. 11.2 and later are unchanged. - **A Timeline enumeration the widget hides is refused when set to a non-default value, instead of failing the build** (ako/mxcli#1762, mendixlabs/mxcli#1668) — a hidden pluggable-widget property must hold its default, or mx check reports CE0463 "The definition of this widget has changed". `mxcli check` already refused this for the one Timeline hide rule it could read (`groupEvents: false`), but it read only 2 of the 6 (`widget describe timeline`: "2 of 6 editor hide-rules recognized"). So `customVisualization: true, groupByDayOptions: 'dayMonth'` and `groupByKey: 'day', groupByMonthOptions: 'monthYear'` passed `check` and `exec` and then failed mx check. The editorConfig's other rules sit in shapes the extractor did not read: a `switch` on `groupByKey` with `case`/`default` arms, and a ternary that terser folds into the `switch(…)` head, where the head's paren was taken for a conditional group. Both are read now, as are fall-through and grouped labels, and an `if(…)` head written the same way (HeatMap's `playground`). Timeline is 6 of 6 (3.1.1, 3.2.2 and 3.2.3), and each hidden enumeration gets the verdict `groupEvents` already had: an MDL-WIDGET10 **error** when set to anything but its default, which `exec` refuses before writing; a warning at its default; written at its default when left out. `describe` no longer prints a hidden property that holds its default, so its own output re-checks without warnings. MDL-WIDGET10 now says "is none of "day", "month"" for a default arm instead of "matches". The warning the reporter saw on hidden `title`/`description`/`timeIndication` came from a hand-written fallback table that listed only the text templates of that same hide. Measured on Mendix 11.13.0 with Timeline 3.2.2, one widget per variant: all six hidden variants were CE0463 before, all three controls 0 errors, and `check` now refuses exactly those six. The corrected pages build with 0 errors under `--no-update-widgets`, and describe → exec reports them unchanged. The File Uploader shape of ako/mxcli#258 (a ternary-wrapped `hidePropertiesIn` gating `createImageAction`/`createFileAction` on `uploadMode`) was already read via the else-if chain support of mendixlabs/mxcli#1583, and is now pinned by a test against File Uploader 2.5.0. - **An activity edited by `create or modify` keeps its element IDs** (ako/mxcli#1761, mendixlabs/mxcli#1667) — re-running `create or modify microflow|nanoflow` with only a filter's expression changed replaced the activity: the activity, its action and its operation all got new `$ID`s, while a decision whose condition changed the same way kept its own. `check`, `mx check`, `diff` and `describe` all looked fine, but Studio Pro compares and merges by `$ID`, so it showed the filter as one activity removed and another added, and a teammate's edit of the same activity on another branch became a conflict. An activity replaced by one of the same kind (the same action) is now edited in place: it keeps its `$ID`, its place in the flow and its size, and so do its action and every element under it that pairs up — a member change by its attribute or association, an argument by its parameter, a nested element by its key and type. A removed member's `$ID` goes onto no other member; a different operation (a `find` where a `filter` was) is new. This covers a filter or find expression, a retrieve's `where`, change and create member values, call arguments and any other same-kind edit, at top level, in a loop body and on an activity with an error handler, and `alter … replace` by one activity of the same kind does the same. Several adjacent activities each changed in their own place are each edited so; before, the run was replaced as a whole. `exec` now tells the two outcomes apart: `spliced: 1 edited in place` for a node that kept its `$ID` (an activity, a decision's condition, a return's value), `replaced with new element IDs` for one that did not. diff --git a/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl b/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl index dff335a7a..31bb7b710 100644 --- a/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl +++ b/mdl-examples/bug-tests/1771-alter-user-task-targeting.mdl @@ -15,6 +15,7 @@ mdl 1; -- 9.24.30.51827, 10.24.24.119653 before: the old targeting was kept. After: -- `[Name = 'reviewer']` and ACT_GetReviewers. 0 errors, with -- the version's key set (Workflows$UserTask on 9.24). +-- 10.11.0.37074 after: the new targeting, 0 errors. -- 11.13.0 the new targeting both before and after, 0 errors. create module Probe1771At; diff --git a/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl b/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl index bf4fe380e..d5c3934eb 100644 --- a/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl +++ b/mdl-examples/bug-tests/1771-user-task-below-10-12.mdl @@ -25,6 +25,10 @@ mdl 1; -- describe → exec prints "Unchanged workflow". Converted up -- to 10.24.24, the tasks are those mxcli writes on 10.24 -- directly. +-- 10.11.0.37074 before: TypeCacheUnknownTypeException. After: 0 errors, +-- each multi-user task's completion with AwaitAllUsers (10.2+). +-- `mx convert -p` 10.11 changes nothing. describe → exec +-- prints "Unchanged workflow". -- 10.24.24, 11.13.0 0 errors, with the same key sets as before the fix. create module Probe1771Ut; diff --git a/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl b/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl index f84d8326f..d7bf181e9 100644 --- a/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl +++ b/mdl-examples/bug-tests/1771-user-task-features-by-version.mdl @@ -19,21 +19,18 @@ mdl 1; -- EXPECTED: `mxcli check` passes this file. With -p on a project below a -- floor, each workflow that states the feature is reported as MDL-VERSION01, -- naming it. Measured on fresh apps: --- 9.24.30.51827 WF_Majority, WF_Await, WF_Microflow, WF_Groups --- 10.24.24.119653 WF_Groups +-- 9.24.30.51827 WF_Majority, WF_Await, WF_Microflow, WF_Groups; exec +-- writes nothing. +-- 10.11.0.37074 WF_Groups. Without it, exec writes the other four as +-- Workflows$UserTask with a MultiInputCompletion, and mx +-- check reports 0 errors. +-- 10.24.24.119653 WF_Groups. -- 11.13.0 none; exec writes all five, and mx check reports 0 errors. create module Probe1771Gt; create persistent entity Probe1771Gt.Request ( Title: String(100) ); -create microflow Probe1771Gt.ACT_Groups ($Workflow: System.Workflow, $Context: Probe1771Gt.Request) -returns list of System.WorkflowGroup as $Groups -begin - retrieve $Groups from System.WorkflowGroup; - return $Groups; -end; - create microflow Probe1771Gt.ACT_Decide ($WorkflowUserTask: System.WorkflowUserTask) returns String as $Outcome begin @@ -93,6 +90,6 @@ create workflow Probe1771Gt.WF_Groups begin user task Review 'Review' page Probe1771Gt.TaskPage - targeting groups microflow Probe1771Gt.ACT_Groups + targeting groups xpath [Name = 'Reviewers'] outcomes 'Done' { }; end workflow; diff --git a/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl b/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl index c23a49cc1..86737d195 100644 --- a/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl +++ b/mdl-examples/bug-tests/1771-user-task-targeting-below-11-2.mdl @@ -26,6 +26,8 @@ mdl 1; -- 10.24.24.119653 before: CE1859 on every task. After: 0 errors. -- `mx convert -p` 10.24.24 changes no key and no value. -- describe → exec prints "Unchanged workflow". +-- 10.11.0.37074 0 errors. The task is a Workflows$UserTask there (see +-- 1771-user-task-below-10-12.mdl), with the same UserSource. -- 11.13.0 0 errors before and after, with the same key sets. create module Probe1771Tg; From f62863bc653bb2ebccf6a4192d554f34c34ba135 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 29 Oct 2026 09:59:53 +0000 Subject: [PATCH 41/44] ci: add Mendix 11.16.0 to the nightly test matrix Mendix 11.16.0 is published on the CDN (mxbuild-11.16.0.tar.gz, 27 Oct 2026) but was not exercised by CI. Added as a seventh entry rather than replacing an older one: 11.13.0 is the measured floor for timer-started workflow event sub-processes, and 11.15.0 is the first version with the MessageDefinition2 document shape (ako/mxcli#987), so each keeps a job running at its own gate. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011oK82teqHJLFTo5pLuHMt1 --- .github/workflows/nightly.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index da3b5719e..6ea88b119 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -26,6 +26,7 @@ jobs: - '11.13.0' - '11.14.0' - '11.15.0' + - '11.16.0' fail-fast: false name: test (Mendix ${{ matrix.mendix-version }}) steps: From 2788811a1437b4e28874714a8f282cb41c6c7044 Mon Sep 17 00:00:00 2001 From: engalar Date: Fri, 30 Oct 2026 10:53:56 +0800 Subject: [PATCH 42/44] =?UTF-8?q?fix(git):=20the=20git-merge=20fixtures=20?= =?UTF-8?q?run=20git=20hermetically=20and=20show=20what=20the=20merge=20sa?= =?UTF-8?q?id=20=E2=80=94=20CI's=20modelConflictRepo=20saw=20no=20unresolv?= =?UTF-8?q?ed=20files=20on=20ubuntu,=20and=20the=20fixture=20threw=20the?= =?UTF-8?q?=20merge's=20output=20away?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- cmd/mxcli/gitdoctor_test.go | 10 ++++++++++ cmd/mxcli/gitmerge_test.go | 10 +++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/cmd/mxcli/gitdoctor_test.go b/cmd/mxcli/gitdoctor_test.go index a621e2569..aeb61e2fd 100644 --- a/cmd/mxcli/gitdoctor_test.go +++ b/cmd/mxcli/gitdoctor_test.go @@ -177,10 +177,20 @@ func TestGitDoctor_SSHURLFormRewrittenByFix(t *testing.T) { // doctorNoteRepo makes a two-commit repository holding an MPR v2 project: // the base commit, then a page retitle. Returns dir and the .mpr path. +// +// The fixture's git is hermetic: no system or global config leaks in from the +// machine this runs on (runner images, developer setups and CI images differ), +// bytes are committed exactly as written, and no repository above the temp +// dir is discovered. Without this, git-merge fixtures see a different merge +// on each machine and fail with the merge's own evidence thrown away. func doctorNoteRepo(t *testing.T) (string, string) { t.Helper() dir := t.TempDir() + t.Setenv("GIT_CONFIG_GLOBAL", os.DevNull) + t.Setenv("GIT_CONFIG_NOSYSTEM", "1") + t.Setenv("GIT_CEILING_DIRECTORIES", filepath.Dir(dir)) gitIn(t, dir, "init", "-q", "-b", "feature") + gitIn(t, dir, "config", "core.autocrlf", "false") units := baseUnits(t) writeProject(t, dir, "11.13.0", units) gitIn(t, dir, "add", "-A") diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index 668feed6d..2a905d783 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -40,12 +40,12 @@ func modelConflictRepo(t *testing.T) (string, string) { gitIn(t, dir, "add", "-A") gitIn(t, dir, "commit", "-q", "-m", "side change") gitIn(t, dir, "switch", "-q", "feature") - _, err := gitInE(t, dir, "merge", "side") + mergeOut, err := gitInE(t, dir, "merge", "side") if err != nil && !strings.Contains(err.Error(), "exit status") { - t.Fatalf("fixture: expected a conflicted merge, got %v", err) + t.Fatalf("fixture: expected a conflicted merge, got %v\n%s", err, mergeOut) } if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); !strings.Contains(uu, ".mxunit") { - t.Fatalf("fixture: expected unresolved model files, got %q", uu) + t.Fatalf("fixture: expected unresolved model files, got %q\nmerge said:\n%s", uu, mergeOut) } return dir, mprPath } @@ -70,9 +70,9 @@ func twoUnitConflictRepo(t *testing.T) (string, string) { gitIn(t, dir, "add", "-A") gitIn(t, dir, "commit", "-q", "-m", "side change") gitIn(t, dir, "switch", "-q", "feature") - _, err := gitInE(t, dir, "merge", "side") + mergeOut, err := gitInE(t, dir, "merge", "side") if err != nil && !strings.Contains(err.Error(), "exit status") { - t.Fatalf("fixture: expected a conflicted merge, got %v", err) + t.Fatalf("fixture: expected a conflicted merge, got %v\n%s", err, mergeOut) } return dir, mprPath } From 69a1fdf516b9c8d333c9cba20cfd144a7dd4acd2 Mon Sep 17 00:00:00 2001 From: engalar Date: Fri, 30 Oct 2026 11:47:31 +0800 Subject: [PATCH 43/44] fix(git): the model merge runs mx merge over three project folders, gates every resolution on hashes, and aborts cleanly on every failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ako's review of #1674: --ours/--theirs took a whole side including the .mpr, so the other side's added units became files no longer in the project and deleted ones stayed listed with no file; --plan refused exactly that case and its error paths left MERGE_HEAD behind (Studio Pro re-runs the half-done merge on every open); every non-zero engine exit was read as a merge decision, so a crash (exit 129) wrote a plan and pointed at side-taking; and mx git-merge's file form does not run on Linux at all (its git lookup is Windows-bound). - the engine is called as mx merge BASE MINE THEIRS over the three project folders (Linux devcontainers included) through docker.PrepareMxCommand, and it owns the .mpr: its result carries both sides' document sets, so a unit one side added while the other changed existing ones is kept - exit 2 alone is a decision (report + plan template + clean abort); any other exit, and an mx that will not start, is reported as the engine failing and aborted cleanly - --ours/--theirs stay but their commit is hash-gated: when the sides' document sets differ the commit is refused (orphan/missing units) and the merge aborts — a whole side never silently drops documents - every --plan refusal aborts the merge (stale already did; invalid take, unreadable snapshots and the hash refusal now do too) - a merge already in progress is resumed instead of re-run, so the mixed-conflict advice (resolve text files, git add, re-run) works - .merge-plan.json is junk for mxcli git commit --- .claude/skills/mendix/teamserver-git/SKILL.md | 8 +- CHANGELOG.md | 2 +- cmd/mxcli/cmd_git_commit.go | 4 + cmd/mxcli/cmd_git_merge.go | 282 ++++++++++++------ cmd/mxcli/gitmerge_test.go | 204 ++++++++++++- docs-site/src/tools/outside-studio-pro.md | 45 +-- .../2026-10-30-git-merge-mx-folder-mode.md | 57 ++++ 7 files changed, 473 insertions(+), 129 deletions(-) create mode 100644 docs/plans/2026-10-30-git-merge-mx-folder-mode.md diff --git a/.claude/skills/mendix/teamserver-git/SKILL.md b/.claude/skills/mendix/teamserver-git/SKILL.md index ea51485aa..944bff868 100644 --- a/.claude/skills/mendix/teamserver-git/SKILL.md +++ b/.claude/skills/mendix/teamserver-git/SKILL.md @@ -117,11 +117,13 @@ git push origin refs/notes/mx_metadata is a binary conflict), and a textual merge of `mprcontents/` leaves an index that does not match the files. Model-aware merging exists in two forms: **`mxcli git merge `** from the command line (it hands the - conflict to `mx git-merge`, Studio Pro's own engine, and commits the - resolved model as a normal merge commit with the note), and **Version + conflict to `mx merge` over three project folders — Studio Pro's own + engine, Linux devcontainers included — and commits the resolved model as a + normal merge commit with the note), and **Version Control › Merge Changes Here** in Studio Pro (*Merge feature branch* or *Cherry Pick*) — which is also where the conflicts mx cannot auto-resolve - go. A conflict `mxcli git merge` cannot resolve aborts the merge cleanly. + go. A conflict `mxcli git merge` cannot resolve — and an engine failure — + aborts the merge cleanly; nothing is left half-done. Studio Pro's history on Team Server is linear for a reason: do not "fix" a conflicted model merge by hand-editing files. - **No `--amend` or force-push of a pushed commit.** Before pushing, an amend is diff --git a/CHANGELOG.md b/CHANGELOG.md index e57c8e024..c7ea8705c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added - **`mxcli git commit -m … -p app.mpr` — commit the way Studio Pro does** (mendixlabs/mxcli#1669) — Studio Pro's *Commit* is more than `git commit`: it keeps the ContentsHash index in step, stages exactly the files a Mendix repository ships, and attaches the mx_metadata note. The new command is that sequence in one line: mismatched hashes are repaired from the files on disk first (`MISSING` and `ORPHAN` entries have no automatic repair and refuse the commit), everything is staged and the files Studio Pro never commits (`project-settings.user.json`, `*.mpr.lock`, `*.mpr.bak`, and the project's own `.mendix-cache/`, `deployment/`, `theme-cache/`, `.mxcli/` beside the `.mpr` — a folder of the same name further down, such as `resources/deployment/`, is committed) are unstaged again, `git commit` runs with the caller's message and identity, and the mx_metadata note is computed from the new commit and attached, so Studio Pro can version-check the revision and show what changed. `--push` pushes the branch and the notes ref together (`git push` alone loses the note the next time Studio Pro fetches); `--dry-run` reports what it would commit without committing — and without repairing anything, hash drift included, because a dry run writes nothing: it stages into a copy of the index, so what the user had staged is left as it was. The commit is refused while Studio Pro has the project open (`.mpr.lock`) — honouring the same `MXCLI_ALLOW_STUDIO_PRO_OPEN=1` override as every write — and during an unresolved merge, which `git add -A` would otherwise commit with its conflict markers. Docs: "Working Outside Studio Pro" › *One commit, all the steps*. -- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx git-merge BASE MINE THEIRS BASE_COMMIT MINE_COMMIT THEIRS_COMMIT`. `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three projects to `mx git-merge`, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything mx auto-resolves is staged and committed as the normal two-parent merge commit with the mx_metadata note — measured end-to-end on a two-sided domain-model change, `AutoResolved`, the merge commit held both sides and the note matched Studio Pro's own merge in shape; a conflict mx reports as `ManualResolutionRequired` is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit (ours and theirs, overlapping units marked); the run also writes a plan template (`.merge-plan.json`) beside the project, and the human then re-runs with `--plan ` to resolve **per unit** — each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files (modified-on-both units only; added/deleted/moved goes to Studio Pro) — or with `--ours`/`--theirs` to take one side wholesale, or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; non-model conflicts (text sources) are left in place like plain git. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. +- **`mxcli git merge ` — model-aware merging from the command line** (mendixlabs/mxcli#1669) — `git merge` cannot merge Mendix models: the `.mpr` is SQLite and every `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` (measured in the git lab: both files, every time). Studio Pro resolves that with a model-aware merge, and the same engine ships as a command — `mx merge BASE MINE THEIRS` over three project folders, which runs on Linux devcontainers too (the older `mx git-merge` file form is Windows-bound). `mxcli git merge ` runs `git merge` first (a merge git resolves itself is annotated like `mxcli git commit`; a merge stopped on text conflicts resumes once those are resolved and staged), and on a model-file conflict checks the base and theirs revisions out into temporary worktrees and hands the three project folders to the engine, mx resolved for the project's Mendix version (PATH, the mxcli cache, the Studio Pro installation): everything the engine auto-resolves is staged, hash-checked and committed as the normal two-parent merge commit with the mx_metadata note — and the engine owns the `.mpr`, so a unit one side added while the other changed existing ones is kept, not dropped; a conflict the engine reports as `ManualResolutionRequired` (exit 2) is a review step, never a silent choice: the merge aborts leaving the working tree exactly as before, and both sides' changes are printed unit by unit at property level (from `mx diff`), with a plan template (`.merge-plan.json`, never staged by `mxcli git commit`) beside the project — the human re-runs with `--plan ` to resolve **per unit** (each conflicted unit takes `ours` or `theirs`, the `.mpr` index is repaired from the mixed files; modified-on-both units only, added/deleted/moved goes to Studio Pro), or with `--ours`/`--theirs` to take one side of every conflicted unit — hash-gated: when the sides' document sets differ (one side added or deleted a unit) the commit is refused and the merge aborted, because a whole side would orphan the other side's documents — or resolves in Studio Pro (*Version Control › Merge Changes Here*), which remains the tool for mixed resolutions; any other engine exit (its 129, an mx that will not start) is reported as the engine failing and the merge is aborted cleanly, never steered into a decision. Non-model conflicts (text sources) are left in place like plain git: resolve them, `git add` them, re-run the command. Refused while Studio Pro has the project open. Docs: "Working Outside Studio Pro" › *Merging branches*. - **`mxcli git doctor [-p app.mpr] [--fix]` — audit a repository for everything a plain-git history misses, and repair it** (mendixlabs/mxcli#1669) — a repository committed only with git drifts from what Studio Pro expects in four ways, each invisible on its own: the local git config Studio Pro writes is absent (`core.autocrlf=False` keeps a checkout from rewriting line endings inside `.mxunit` files on Windows; `mendix.commits-since-gc` and `mendix.lineEndingResetDone` are its own bookkeeping), the remote is SSH which Studio Pro cannot use (HTTPS only), commits have no note under `refs/notes/mx_metadata` (history shows `(unknown)`, the version check fails), and a malformed note Studio Pro cannot read at all. `doctor` reports all four in one read-only pass and exits 1 while an issue remains, so it can gate CI. `--fix` sets the config keys, rewrites SSH remote URLs (`git@host:org/app.git`, `ssh://…`) to HTTPS — judged as configured, so a `url..insteadOf` rule in the user's git config (https → `git@` is a common one) is reported rather than "repaired" on every run — and backfills notes: computed from the model with `-p` (the same engine `git note` uses), or a template note with a `--version` ModelerVersion where the model at a commit cannot be read (MPR v1 history, a revision without the `.mpr`); with neither, the newest valid note in history supplies the version. Notes Studio Pro wrote are never touched, with or without `--fix`; `--fix` does not push. A clone whose notes ref lacks the remote's notes — `git clone` never fetches `refs/notes/*`, so every fresh clone and CI checkout — is reported as exactly that, with the fetch command, rather than as a history of unannotated commits; `--fix` fetches the notes first, as a fast-forward only. Docs: "Working Outside Studio Pro" › *Auditing a repository*. ### Fixed diff --git a/cmd/mxcli/cmd_git_commit.go b/cmd/mxcli/cmd_git_commit.go index a51d88bf2..27e575613 100644 --- a/cmd/mxcli/cmd_git_commit.go +++ b/cmd/mxcli/cmd_git_commit.go @@ -261,6 +261,10 @@ func isJunkPath(rel, projDir string) bool { return true case strings.HasSuffix(strings.ToLower(base), ".mpr.lock"), strings.HasSuffix(strings.ToLower(base), ".mpr.bak"): return true + // `mxcli git merge` writes the plan template beside the .mpr for the + // human to edit; it is working material, never project content. + case strings.HasSuffix(strings.ToLower(base), ".merge-plan.json"): + return true } inProject := f if projDir != "." && projDir != "" { diff --git a/cmd/mxcli/cmd_git_merge.go b/cmd/mxcli/cmd_git_merge.go index 8d4679e54..3aeecbe6b 100644 --- a/cmd/mxcli/cmd_git_merge.go +++ b/cmd/mxcli/cmd_git_merge.go @@ -22,53 +22,71 @@ import ( // cmd_git_merge.go: `mxcli git merge ` — merge a branch the way Studio // Pro does. `git merge` runs first; when it stops on the model files (the // .mpr is SQLite and every .mxunit is BSON, so any two-sided model change is -// a binary conflict), the conflict is handed to `mx git-merge` — the same -// model-aware engine Studio Pro's own merge uses — with the three revisions -// checked out into temporary worktrees. What mx resolves is committed as a -// normal two-parent merge commit with the mx_metadata note; what it reports -// as ManualResolutionRequired aborts the merge with the conflict list and a -// pointer to Studio Pro, leaving nothing behind. +// a binary conflict), the three revisions are handed to `mx merge` — the same +// model-aware engine Studio Pro's own merge uses, over the base and theirs +// checkouts and the working tree — with the project's Mendix version's mx. +// What the engine resolves is staged, hash-checked and committed as a normal +// two-parent merge commit with the mx_metadata note; what it reports as +// ManualResolutionRequired (exit 2) aborts the merge with the conflict list +// and a pointer to Studio Pro, leaving nothing behind; any other non-zero +// exit is the engine failing, reported as such, also with a clean abort. var gitMergeCmd = &cobra.Command{ Use: "merge ", - Short: "Merge a branch with model-aware conflict resolution (mx git-merge)", + Short: "Merge a branch with model-aware conflict resolution (mx merge)", Long: `Merge into the current branch, resolving model conflicts the way Studio Pro does. 'git merge ' runs first. A merge git resolves itself (disjoint files, a fast-forward) is committed and annotated like 'mxcli git commit' does. When git stops on the model files — the .mpr and the .mxunit units are binary, so -two sides changing the model always conflict — the three revisions are -checked out into temporary worktrees and 'mx git-merge BASE MINE THEIRS' -(Studio Pro's own model-aware merge, resolved for the project's Mendix -version) decides: - - - everything mx auto-resolves is staged and committed as the merge commit, - with the mx_metadata note attached (this keeps both sides — the usual - best outcome); - - a conflict mx reports as ManualResolutionRequired is never merged - silently: the merge is aborted, and both sides' changes are printed for - review — at property level, from 'mx diff' (e.g. +two sides changing the model always conflict — the base and theirs revisions +are checked out into temporary worktrees and the three project folders go to +'mx merge BASE MINE THEIRS' (Studio Pro's own model-aware merge, resolved for +the project's Mendix version): + + - everything the engine auto-resolves is staged, hash-checked and committed + as the merge commit, with the mx_metadata note attached (this keeps both + sides — the usual best outcome; the engine owns the .mpr, so a unit one + side added while the other changed existing ones is kept, not dropped); + - a conflict the engine reports as ManualResolutionRequired (exit 2) is + never merged silently: the merge is aborted, and both sides' changes are + printed for review — at property level, from 'mx diff' (e.g. "New type > Length: 100 → 150" on ours, "→ 200" on theirs). Then decide: --plan resolve per unit from the plan template written beside the .mpr (.merge-plan.json): set "take" to "ours"/"theirs" per unit; the .mpr index is repaired from the mixed files. Modified-on-both units only — added/deleted/moved conflicts go to Studio Pro. - --ours / --theirs take one side of every conflicted model file - (deterministic; mx is not consulted again) + --ours / --theirs take one side of every conflicted unit + (deterministic; mx is not consulted again). The .mpr is + taken from that side too and the result is hash-checked: + when the sides' document sets differ (one added or + deleted a unit), the commit is refused — a whole side + would leave the other side's units orphaned on disk or + listed with no file — and the merge is aborted. Resolve + those in Studio Pro instead. Studio Pro (Version Control › Merge Changes Here) for mixed within-unit resolutions — still the tool when a decision must mix both sides inside one unit; + - any other engine exit is mx failing, not a decision: it is reported and + the merge is aborted cleanly; - a conflict in non-model files (text sources) is left in place for manual - resolution, like plain git. + resolution, like plain git: resolve them, 'git add' them, and re-run + this command — it finishes the merge (model conflicts included) and + commits. + +--ours/--theirs take one side of every conflicted unit — never a side of the +.mpr without the hash check, because the .mpr lists every document: a whole +side would drop the other side's added units and resurrect its deleted ones. The merge is refused while Studio Pro has the project open (.mpr.lock). MPR v2 projects only. The mx binary is resolved for the project's Mendix version (PATH, the mxcli cache, or the Studio Pro installation).`, Example: ` mxcli git merge feature/orders -p app.mpr mxcli git merge origin/main -p app.mpr --push - mxcli git merge feature/orders -p app.mpr --theirs # after reviewing the report`, + mxcli git merge feature/orders -p app.mpr --theirs # after reviewing the report + mxcli git merge feature/orders -p app.mpr --plan app.mpr.merge-plan.json`, Args: cobra.ExactArgs(1), RunE: runGitMerge, SilenceUsage: true, @@ -105,11 +123,17 @@ func runGitMerge(cmd *cobra.Command, args []string) error { case theirs: strategy = "theirs" } - return mergeRun(project, ref, message, strategy, plan, push, mxRunners{merge: realMxGitMerge, diff: realMxDiff}, cmd.OutOrStdout()) + return mergeRun(project, ref, message, strategy, plan, push, mxRunners{merge: realMxMerge, diff: realMxDiff}, cmd.OutOrStdout()) } // mxRunners are the mx invocations, injected for tests. -type mxMergeRunner func(base, mine, theirs string, shas [3]string) (int, string, error) +// +// merge runs the engine over the three PROJECT FOLDERS — the base checkout, +// the working tree, the theirs checkout — and writes its result into the +// working tree, App.mpr included. Exit 0 is a resolved merge, 2 is a decision +// the engine could not make (its per-unit list is on stdout), anything else +// is the engine failing. +type mxMergeRunner func(baseDir, mineDir, theirsDir string, shas [3]string) (int, string, error) type mxDiffRunner func(base, mine, outJSON string) error type mxRunners struct { @@ -117,19 +141,20 @@ type mxRunners struct { diff mxDiffRunner } -// mergeRun is the command without its wiring. runMx is injected for tests. -// mergeRun is the command without its wiring. runMx is injected for tests. +// mergeRun is the command without its wiring. runners are injected for tests. // // Conflict handling is a three-step flow a human stays in charge of: // // 1. `git merge` runs; a merge it resolves itself is annotated and done. -// 2. A model conflict goes to `mx git-merge` first — what it auto-resolves -// keeps both sides, which is almost always the best outcome — and is -// committed with the note. -// 3. What mx reports as ManualResolutionRequired is never merged silently: -// with no strategy the merge is aborted and both sides' changes are -// printed for review; `--ours`/`--theirs` then take one side wholesale -// (mx is not consulted — the human has decided). +// 2. A model conflict goes to `mx merge` over the three project folders +// first — what the engine auto-resolves keeps both sides, which is almost +// always the best outcome — and is committed with the note. +// 3. What the engine reports as ManualResolutionRequired (exit 2) is never +// merged silently: with no strategy the merge is aborted and both sides' +// changes are printed for review; `--plan` then resolves per unit, and +// `--ours`/`--theirs` take one side of every conflicted unit, hash-gated +// so a side that would drop or orphan documents is refused. Any other +// engine exit is a failure, not a decision: reported, aborted cleanly. func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners mxRunners, out io.Writer) error { switch strategy { case "", "ours", "theirs": @@ -161,20 +186,24 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return applyPlan(root, repo, abs, mprPath, ref, plan, message, branch, push, out) } - mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) - if mergeErr == nil { - if strings.Contains(mergeOut, "Already up to date") { - fmt.Fprintln(out, "Already up to date.") - return nil - } - head := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) - fmt.Fprintf(out, "Merged %s without model conflicts (%s).\n", ref, shortCommit(head)) - return finishMerge(out, repo, head, branch, push) - } - - // A ref that does not resolve fails before any conflict exists. + // A merge already in progress is resumed, not re-run: a previous run may + // have stopped on text conflicts, which the human resolved and staged + // since. Re-running git merge would only refuse ("not concluded"). if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { - return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + if mergeErr == nil { + if strings.Contains(mergeOut, "Already up to date") { + fmt.Fprintln(out, "Already up to date.") + return nil + } + head := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) + fmt.Fprintf(out, "Merged %s without model conflicts (%s).\n", ref, shortCommit(head)) + return finishMerge(out, repo, head, branch, push) + } + // A ref that does not resolve fails before any conflict exists. + if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { + return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + } } conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) @@ -187,7 +216,7 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m } } if len(other) > 0 { - fmt.Fprintln(out, "Non-model conflicts, resolve by hand, then 'git add' and 'git commit':") + fmt.Fprintln(out, "Non-model conflicts, resolve by hand, then 'git add' them and re-run this command to resolve the model and commit:") for _, p := range other { fmt.Fprintf(out, " %s\n", p) } @@ -198,27 +227,34 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m oursSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) theirsSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "MERGE_HEAD")) - // The human has decided: take one side of every conflicted model file. - // Deterministic, and mx is not consulted — `git checkout --ours/--theirs` - // reads the conflicted stages directly. + // The human has decided: take one side of every conflicted model file. mx + // is not consulted — `git checkout --ours/--theirs` reads the conflicted + // stages directly. The .mpr is taken from the side too, and the result is + // hash-checked before committing: a whole side includes the .mpr's list of + // documents, so when the sides' document sets differ (one side added or + // deleted a unit) committing would orphan the other side's units or list + // units with no file. That refusal aborts the merge — it never commits a + // half-project. if strategy != "" { side := "--" + strategy for _, p := range model { if _, err := gitOutIn(root, nil, "checkout", side, "--", p); err != nil { - return fmt.Errorf("take the %s version of %s: %w", strategy, p, err) + return abortMerge(root, fmt.Errorf("take the %s version of %s: %w", strategy, p, err)) } } - args := []string{"add", "--"} - args = append(args, model...) - if _, err := gitOutIn(root, nil, args...); err != nil { - return fmt.Errorf("stage the resolved model files: %w", err) + if err := repairOrRefuse(root, abs, out); err != nil { + return err } return commitTheMerge(root, repo, model, message, branch, push, out) } - // mx first: what it auto-resolves keeps both sides, which is almost - // always better than taking one side wholesale. - // Check out the base and theirs revisions; MINE is the working tree. + // The engine first: what it auto-resolves keeps both sides, which is + // almost always better than taking one side wholesale. The base and + // theirs revisions are checked out as folders; MINE is the working tree. + // The engine writes its result into the working tree and owns the .mpr — + // its unit list carries both sides' added and deleted documents, so a + // unit one side added while the other changed existing ones is kept, not + // dropped. tmp, err := os.MkdirTemp("", "mxcli-gitmerge-") if err != nil { return err @@ -243,10 +279,13 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m version := projectMendixVersion(mprPath) fmt.Fprintf(out, "Model files conflict (%s); merging the model with mx %s…\n", strings.Join(shortNames(model), ", "), version) + // Folder mode: the engine takes the three project folders and writes its + // result into MINE — the working tree. + projRel := filepath.FromSlash(filepath.Dir(repo.MPRPath)) // "." at the repo root code, mxOut, runErr := runners.merge( - filepath.Join(baseDir, filepath.FromSlash(repo.MPRPath)), - abs, - filepath.Join(theirsDir, filepath.FromSlash(repo.MPRPath)), + filepath.Join(baseDir, projRel), + filepath.Dir(abs), + filepath.Join(theirsDir, projRel), [3]string{baseSha, oursSha, theirsSha}, ) if s := strings.TrimSpace(mxOut); s != "" { @@ -254,9 +293,16 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m } if runErr != nil { cleanWorktrees() - return fmt.Errorf("mx git-merge failed: %w", runErr) + return abortMerge(root, fmt.Errorf("mx merge failed to run: %w", runErr)) + } + if code != 0 && code != 2 { + // A crash, an unreadable base, anything but a merge decision: report + // the engine as failing and leave nothing behind. Only exit 2 asks + // for a decision. + cleanWorktrees() + return abortMerge(root, fmt.Errorf("mx merge failed (exit %d); the merge is aborted — if it persists, resolve in Studio Pro (Version Control › Merge Changes Here)", code)) } - if code != 0 { + if code == 2 { // The review step: what each side changed, at property level (mx // diff), plus a per-unit plan template. Then leave nothing behind — // the human decides. @@ -279,14 +325,20 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return fmt.Errorf("the model merge has conflicts and aborting it failed (%v); resolve in Studio Pro", abortErr) } fmt.Fprintln(out, "Merge aborted; the working tree is back to the pre-merge state.") - return errors.New("the model merge needs a decision: re-run with --plan to resolve per unit, --ours/--theirs to take one side wholesale, or resolve in Studio Pro (Version Control › Merge Changes Here)") + return errors.New("the model merge needs a decision: re-run with --plan to resolve per unit, or resolve in Studio Pro (Version Control › Merge Changes Here)") } + // Exit 0: the engine resolved the model into the working tree, .mpr + // included — both sides' document sets are in it. Stage, gate, commit. args := []string{"add", "--"} args = append(args, model...) if _, err := gitOutIn(root, nil, args...); err != nil { cleanWorktrees() - return fmt.Errorf("stage the resolved model files: %w", err) + return abortMerge(root, fmt.Errorf("stage the resolved model files: %w", err)) + } + if err := repairOrRefuse(root, abs, out); err != nil { + cleanWorktrees() + return err } if err := commitTheMerge(root, repo, nil, message, branch, push, out); err != nil { cleanWorktrees() @@ -296,6 +348,34 @@ func mergeRun(mprPath, ref, message, strategy, plan string, push bool, runners m return nil } +// abortMerge backs a failed resolution out entirely — working tree, index and +// MERGE_HEAD return to the pre-merge state — and returns the original error. +// Every refusal on this command ends here: a half-done merge makes Studio Pro +// re-run it (and choke) every time the project opens. +func abortMerge(root string, err error) error { + if _, abortErr := gitOutIn(root, nil, "merge", "--abort"); abortErr != nil { + return fmt.Errorf("%w (aborting the merge failed too: %v — resolve it in Studio Pro)", err, abortErr) + } + return err +} + +// repairOrRefuse is the hash gate every side- or plan-based resolution ends +// with: ContentsHash values are repaired from the files on disk (the files +// are the truth once units have been taken from both sides), and a MISSING or +// ORPHAN entry — a unit the .mpr lists with no file behind it, or a file no +// .mpr entry lists — refuses the commit: committing it would drop or orphan +// documents. The refusal aborts the merge. +func repairOrRefuse(root, mprPath string, out io.Writer) error { + err := fixHashes(mprPath, true, out) + if err == nil { + return nil + } + if !errors.Is(err, errHashIssuesRemain) { + return abortMerge(root, err) + } + return abortMerge(root, errors.New("refusing to commit: the ContentsHash index has MISSING or ORPHAN entries (see above) — the sides' document sets differ, and committing would drop or orphan units; resolve in Studio Pro (Version Control › Merge Changes Here)")) +} + // commitTheMerge commits the resolved merge (MERGE_HEAD is present), writes // the note and pushes. stagedPaths, when given, are staged first. func commitTheMerge(root string, repo *gitNoteRepo, stagedPaths []string, message, branch string, push bool, out io.Writer) error { @@ -370,15 +450,19 @@ func projectMendixVersion(mprPath string) string { return v } -// realMxGitMerge shells out to `mx git-merge -v …`, resolved for the -// project's Mendix version. -func realMxGitMerge(base, mine, theirs string, shas [3]string) (int, string, error) { - version := projectMendixVersion(mine) +// realMxMerge shells out to `mx merge BASE MINE THEIRS` over the three +// project folders, mx resolved for the project's Mendix version. +func realMxMerge(baseDir, mineDir, theirsDir string, shas [3]string) (int, string, error) { + version := "" + if matches, _ := filepath.Glob(filepath.Join(mineDir, "*.mpr")); len(matches) > 0 { + version = projectMendixVersion(matches[0]) + } mxPath, err := docker.ResolveMxForVersion("", version) if err != nil { return 1, "", fmt.Errorf("no mx for Mendix %s: %w (model conflicts need it)", version, err) } - cmd := exec.Command(mxPath, "git-merge", "-v", base, mine, theirs, shas[0], shas[1], shas[2]) + cmd := exec.Command(mxPath, "merge", baseDir, mineDir, theirsDir) + docker.PrepareMxCommand(cmd) b, err := cmd.CombinedOutput() code := 0 if err != nil { @@ -521,9 +605,11 @@ func writePlanTemplate(mprPath, baseSha, oursSha, theirsSha string, model []stri return os.WriteFile(mprPath+".merge-plan.json", b, 0o644) } -// applyPlan re-runs the merge (it must conflict the same way again), checks -// the plan against those revisions, resolves per unit, repairs the .mpr index -// from the mixed files and commits — with the note. +// applyPlan re-runs the merge unless one is already in progress (it must +// conflict the same way), checks the plan against those revisions, resolves +// per unit, repairs the .mpr index from the mixed files and commits — with +// the note. Every refusal after the merge started aborts it: a half-done +// merge makes Studio Pro re-run it (and choke) on every open. func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, message, branch string, push bool, out io.Writer) error { raw, err := os.ReadFile(planPath) if err != nil { @@ -539,19 +625,22 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m return fmt.Errorf("plan %s is not valid JSON: %w", planPath, err) } - mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) - if mergeErr == nil { - return errors.New("the merge no longer conflicts — the plan is unnecessary; nothing was done") - } + // An unfinished merge (text conflicts resolved and staged, say) is + // resumed; otherwise the merge runs and must conflict the same way. if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { - return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + mergeOut, mergeErr := gitOutIn(root, nil, "merge", "--no-edit", ref) + if mergeErr == nil { + return errors.New("the merge no longer conflicts — the plan is unnecessary; nothing was done") + } + if _, mhErr := gitOut(root, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); mhErr != nil { + return fmt.Errorf("git merge %s: %s", ref, firstLineOf(mergeOut)) + } } baseSha := strings.TrimSpace(gitOutOr(root, "merge-base", "HEAD", "MERGE_HEAD")) oursSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "HEAD")) theirsSha := strings.TrimSpace(gitOutOr(root, "rev-parse", "MERGE_HEAD")) if plan.Base != baseSha || plan.Ours != oursSha || plan.Theirs != theirsSha { - _, _ = gitOutIn(root, nil, "merge", "--abort") - return errors.New("stale plan: the merge revisions moved since it was written; delete it and re-run the merge to regenerate") + return abortMerge(root, errors.New("stale plan: the merge revisions moved since it was written; delete it and re-run the merge to regenerate")) } conflicted := nulFields(gitOutOr(root, "diff", "--name-only", "--diff-filter=U", "-z")) @@ -566,25 +655,25 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m // (added on one side, deleted, moved) only Studio Pro can decide. baseSnap, err := repo.snapshot(baseSha) if err != nil { - return err + return abortMerge(root, err) } oursSnap, err := repo.snapshot(oursSha) if err != nil { - return err + return abortMerge(root, err) } theirsSnap, err := repo.snapshot(theirsSha) if err != nil { - return err + return abortMerge(root, err) } oursStatus := changeStatuses(computeModelChanges(baseSnap, oursSnap, nil)) theirsStatus := changeStatuses(computeModelChanges(baseSnap, theirsSnap, nil)) take := map[string]string{} for _, u := range plan.Units { if u.Take != "ours" && u.Take != "theirs" { - return fmt.Errorf("plan: unit %s has take %q (ours|theirs)", shortCommit(u.ID), u.Take) + return abortMerge(root, fmt.Errorf("plan: unit %s has take %q (ours|theirs)", shortCommit(u.ID), u.Take)) } if oursStatus[u.ID] != "Modified" || theirsStatus[u.ID] != "Modified" { - return fmt.Errorf("plan: unit %s is %s on ours and %s on theirs — only modified-on-both conflicts can be planned; resolve it in Studio Pro", shortCommit(u.ID), statusOrNone(oursStatus[u.ID]), statusOrNone(theirsStatus[u.ID])) + return abortMerge(root, fmt.Errorf("plan: unit %s is %s on ours and %s on theirs — only modified-on-both conflicts can be planned; resolve it in Studio Pro", shortCommit(u.ID), statusOrNone(oursStatus[u.ID]), statusOrNone(theirsStatus[u.ID]))) } take[u.ID] = u.Take } @@ -592,7 +681,7 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m // Ours everywhere, then the planned units from theirs. for _, p := range model { if _, err := gitOutIn(root, nil, "checkout", "--ours", "--", p); err != nil { - return fmt.Errorf("take the ours version of %s: %w", p, err) + return abortMerge(root, fmt.Errorf("take the ours version of %s: %w", p, err)) } } for _, id := range conflictedUnitIDs(model) { @@ -601,23 +690,22 @@ func applyPlan(root string, repo *gitNoteRepo, mprAbs, mprPath, ref, planPath, m } p := path.Join(repo.Contents, id[:2], id[2:4], id+".mxunit") if _, err := gitOutIn(root, nil, "checkout", "--theirs", "--", p); err != nil { - return fmt.Errorf("take the theirs version of %s: %w", p, err) + return abortMerge(root, fmt.Errorf("take the theirs version of %s: %w", p, err)) } fmt.Fprintf(out, " %s: theirs\n", shortCommit(id)) } // The .mpr index is ours; the mixed unit files have different bytes, so - // the hashes are rewritten from the files — files are the truth. - if err := fixHashes(mprPath, true, out); err != nil { - if !errors.Is(err, errHashIssuesRemain) { - return err - } - return errors.New("refusing to commit: the ContentsHash index has MISSING or ORPHAN entries (see above)") + // the hashes are rewritten from the files — files are the truth. The + // gate refuses (and aborts) when the sides' document sets differ: a unit + // theirs added is not in ours' .mpr, and committing would orphan it. + if err := repairOrRefuse(root, mprPath, out); err != nil { + return err } args := []string{"add", "--"} args = append(args, model...) if _, err := gitOutIn(root, nil, args...); err != nil { - return fmt.Errorf("stage the resolved model files: %w", err) + return abortMerge(root, fmt.Errorf("stage the resolved model files: %w", err)) } return commitTheMerge(root, repo, nil, message, branch, push, out) } @@ -637,14 +725,16 @@ func statusOrNone(s string) string { return s } -// realMxDiff shells out to `mx diff BASE MINE OUTPUT` (resolved like git-merge). +// realMxDiff shells out to `mx diff BASE MINE OUTPUT` (resolved like the merge). func realMxDiff(base, mine, outJSON string) error { version := projectMendixVersion(mine) mxPath, err := docker.ResolveMxForVersion("", version) if err != nil { return fmt.Errorf("no mx for Mendix %s: %w", version, err) } - if b, err := exec.Command(mxPath, "diff", base, mine, outJSON).CombinedOutput(); err != nil { + cmd := exec.Command(mxPath, "diff", base, mine, outJSON) + docker.PrepareMxCommand(cmd) + if b, err := cmd.CombinedOutput(); err != nil { return fmt.Errorf("mx diff: %v: %s", err, strings.TrimSpace(string(b))) } return nil diff --git a/cmd/mxcli/gitmerge_test.go b/cmd/mxcli/gitmerge_test.go index 2a905d783..22ec07ba9 100644 --- a/cmd/mxcli/gitmerge_test.go +++ b/cmd/mxcli/gitmerge_test.go @@ -5,6 +5,7 @@ package main import ( "bytes" "encoding/json" + "errors" "fmt" "os" "path/filepath" @@ -19,7 +20,7 @@ func bsonKeyValue(key, value string) bson.E { } // `mxcli git merge ` — merge a branch with model-aware resolution: git -// merge first, and on a model-file conflict `mx git-merge` (the same engine +// merge first, and on a model-file conflict `mx merge` (the same engine // Studio Pro uses). The mx run is injected as runMx so the orchestration is // testable without a Mendix installation; the real runner shells out to mx. @@ -77,6 +78,36 @@ func twoUnitConflictRepo(t *testing.T) (string, string) { return dir, mprPath } +// addConflictRepo is a repository where each side ADDED a different unit: only +// the .mpr conflicts, and the sides' document sets differ — the class a whole +// side must refuse, never commit. +func addConflictRepo(t *testing.T) (string, string) { + t.Helper() + dir, mprPath := doctorNoteRepo(t) + ours := baseUnits(t) + oursAdd := "11111111-0000-0000-0000-000000000009" + ours[oursAdd] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Ours")} + writeProject(t, dir, "11.13.0", ours) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "ours adds a microflow") + gitIn(t, dir, "switch", "-q", "-c", "side", "HEAD~1") + theirs := baseUnits(t) + theirsAdd := "11111111-0000-0000-0000-00000000000a" + theirs[theirsAdd] = tUnit{tMod, unitDoc(t, "Microflows$Microflow", "ACT_Theirs")} + writeProject(t, dir, "11.13.0", theirs) + gitIn(t, dir, "add", "-A") + gitIn(t, dir, "commit", "-q", "-m", "theirs adds a microflow") + gitIn(t, dir, "switch", "-q", "feature") + mergeOut, err := gitInE(t, dir, "merge", "side") + if err != nil && !strings.Contains(err.Error(), "exit status") { + t.Fatalf("fixture: expected a conflicted merge, got %v\n%s", err, mergeOut) + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); !strings.Contains(uu, ".mpr") { + t.Fatalf("fixture: expected the .mpr conflict, got %q\nmerge said:\n%s", uu, mergeOut) + } + return dir, mprPath +} + // unitRel is the mprcontents path of a unit id inside a work tree. func unitRel(id string) string { return "mprcontents/" + id[:2] + "/" + id[2:4] + "/" + id + ".mxunit" @@ -132,25 +163,29 @@ func TestGitMerge_ModelConflictAutoResolvedByMx(t *testing.T) { var gotBase, gotMine, gotTheirs string var gotShas [3]string + var baseHasMpr, theirsHasMpr bool runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { gotBase, gotMine, gotTheirs, gotShas = base, mine, theirs, shas + // The worktrees are removed when the merge finishes; check inside. + _, baseHasMprErr := os.Stat(filepath.Join(base, "App.mpr")) + _, theirsHasMprErr := os.Stat(filepath.Join(theirs, "App.mpr")) + baseHasMpr, theirsHasMpr = baseHasMprErr == nil, theirsHasMprErr == nil return 0, "Merge successful.", nil }} var out bytes.Buffer if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { t.Fatalf("mergeRun: %v\n%s", err, out.String()) } - if !strings.Contains(gotBase, "mx-git-lab") && !strings.Contains(filepath.Base(gotBase), ".mpr") { - // the base path must be an .mpr inside a temporary checkout - if filepath.Ext(gotBase) != ".mpr" { - t.Errorf("runner got base %q, want the base checkout's .mpr", gotBase) - } + // Folder mode: the three args are project folders, and MINE is the + // working tree — the engine writes its result there. + if gotMine != dir { + t.Errorf("runner's mine = %q, want the working project folder %q", gotMine, dir) } - if filepath.Base(gotMine) != "App.mpr" { - t.Errorf("runner's mine = %q, want the working .mpr", gotMine) + if !baseHasMpr { + t.Errorf("runner's base folder %q must hold the project's .mpr", gotBase) } - if filepath.Base(gotTheirs) != "App.mpr" { - t.Errorf("runner's theirs = %q, want the theirs checkout's .mpr", gotTheirs) + if !theirsHasMpr { + t.Errorf("runner's theirs folder %q must hold the project's .mpr", gotTheirs) } if gotShas[0] == "" || gotShas[1] == gotShas[0] || gotShas[2] == gotShas[1] { t.Errorf("shas must be base/ours/theirs distinct: %v", gotShas) @@ -267,7 +302,7 @@ func TestGitMerge_ConflictReportShowsBothSidesAndAborts(t *testing.T) { }} var out bytes.Buffer err := mergeRun(mprPath, "side", "", "", "", false, noMx, &out) - if err == nil || !strings.Contains(err.Error(), "--ours") { + if err == nil || !strings.Contains(err.Error(), "--plan") { t.Fatalf("the report must point at the resolution flags, got %v", err) } report := out.String() @@ -445,3 +480,150 @@ func TestGitMerge_ManualWritesPlanAndPropertyReport(t *testing.T) { } gitInE(t, dir, "merge", "--abort") // clean up the fixture } + +// A whole side of a project includes the .mpr's list of documents. When the +// sides' document sets differ (one side added a unit), taking --theirs would +// leave ours' unit on disk with no .mpr entry: the commit is refused and the +// merge aborted, never silently half-committed. +func TestGitMerge_SideTakingRefusesWhenDocumentSetsDiffer(t *testing.T) { + dir, mprPath := addConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + noMx := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + t.Fatal("--theirs must not consult mx") + return 0, "", nil + }} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "theirs", "", false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "refusing to commit") { + t.Fatalf("--theirs must refuse a side that drops ours' unit, got %v\n%s", err, out.String()) + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the refusal must abort the merge") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } +} + +// Exit 129 is the engine failing, not a merge decision: report it, abort +// cleanly, and never write a plan template. +func TestGitMerge_EngineCrashAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + crash := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 129, "Exception in MergeLib", nil + }} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", "", false, crash, &out) + if err == nil || !strings.Contains(err.Error(), "exit 129") { + t.Fatalf("want the engine-failure report, got %v\n%s", err, out.String()) + } + if strings.Contains(out.String(), "Plan template") { + t.Errorf("a crash must not produce a plan:\n%s", out.String()) + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the merge was not aborted") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } + if _, err := os.Stat(mprPath + ".merge-plan.json"); !os.IsNotExist(err) { + t.Errorf("a crash must not write a plan template: %v", err) + } +} + +// mx not being found or not starting is the engine failing to run: the merge +// aborts with the reason, and nothing is left behind. +func TestGitMerge_EngineStartFailureAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + before := commitSubject(t, dir) + + missing := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + return 0, "", errors.New("no mx for Mendix 11.13.0: not on PATH or in the cache (model conflicts need it)") + }} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", "", false, missing, &out) + if err == nil || !strings.Contains(err.Error(), "failed to run") { + t.Fatalf("want the mx failure, got %v\n%s", err, out.String()) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the merge was not aborted") + } + if got := commitSubject(t, dir); got != before { + t.Errorf("HEAD moved: %q", got) + } +} + +// A plan whose take value is not ours/theirs is refused, and the refusal +// aborts the merge: nothing may be left half-done. +func TestGitMerge_PlanRefusalAborts(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + base := strings.TrimSpace(gitOutT(t, dir, "merge-base", "HEAD", "side")) + ours := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "HEAD")) + theirs := strings.TrimSpace(gitOutT(t, dir, "rev-parse", "side")) + plan := fmt.Sprintf(`{"base":%q,"ours":%q,"theirs":%q,"units":[{"id":%q,"take":"sideways"}]}`, + base, ours, theirs, tDM) + planPath := filepath.Join(dir, "merge-plan.json") + if err := os.WriteFile(planPath, []byte(plan), 0o644); err != nil { + t.Fatal(err) + } + noMx := mxRunners{} + var out bytes.Buffer + err := mergeRun(mprPath, "side", "", "", planPath, false, noMx, &out) + if err == nil || !strings.Contains(err.Error(), "ours|theirs") { + t.Fatalf("want the invalid-take refusal, got %v\n%s", err, out.String()) + } + if _, err := gitOut(dir, "rev-parse", "--verify", "--quiet", "MERGE_HEAD"); err == nil { + t.Error("MERGE_HEAD still present: the refusal must abort the merge") + } + if uu := gitOutT(t, dir, "diff", "--name-only", "--diff-filter=U"); strings.TrimSpace(uu) != "" { + t.Errorf("working tree not restored:\n%s", uu) + } +} + +// Re-entry: with the merge already in progress (a previous run stopped and +// the human resolved what they handle by hand), re-running resolves the model +// and finishes — it does not re-run git merge and die on MERGE_HEAD. +func TestGitMerge_ResumesAMergeAlreadyInProgress(t *testing.T) { + dir, mprPath := modelConflictRepo(t) + doctorEnv(t, dir) + + calls := 0 + runner := mxRunners{merge: func(base, mine, theirs string, shas [3]string) (int, string, error) { + calls++ + return 0, "Merge successful.", nil + }} + var out bytes.Buffer + if err := mergeRun(mprPath, "side", "", "", "", false, runner, &out); err != nil { + t.Fatalf("mergeRun: %v\n%s", err, out.String()) + } + if calls != 1 { + t.Errorf("engine runs = %d, want 1", calls) + } + parents := strings.Fields(gitOutT(t, dir, "rev-list", "--parents", "-n", "1", "HEAD")) + if len(parents) != 3 { + t.Fatalf("want a two-parent merge commit, got %v", parents) + } +} + +// The plan template is working material: `mxcli git commit` never stages it. +func TestJunkPath_CoversMergePlanTemplate(t *testing.T) { + if !isJunkPath("App.mpr.merge-plan.json", ".") { + t.Error("App.mpr.merge-plan.json must be junk for staging") + } + if !isJunkPath("app/Project.mpr.merge-plan.json", "app") { + t.Error("a project-folder plan template must be junk too") + } +} diff --git a/docs-site/src/tools/outside-studio-pro.md b/docs-site/src/tools/outside-studio-pro.md index 0d2f0145b..da43493ea 100644 --- a/docs-site/src/tools/outside-studio-pro.md +++ b/docs-site/src/tools/outside-studio-pro.md @@ -219,7 +219,9 @@ the index, so whatever you had staged is left exactly as it was. `.mxunit` is BSON, so two sides changing the model always meet a binary `CONFLICT (content)` — there is nothing for git's textual merge to work on. Studio Pro's *Merge Changes Here* resolves it with a model-aware merge, and -that engine ships as a command: `mx git-merge BASE MINE THEIRS …`. +that engine ships as a command: `mx merge BASE MINE THEIRS` over three +project folders (it runs on Linux devcontainers too; the older `mx git-merge` +file form is Windows-bound). `mxcli git merge ` puts the two together: @@ -231,25 +233,32 @@ mxcli git merge origin/main -p app.mpr --push `git merge ` runs first; a merge git resolves itself (disjoint files, a fast-forward) is annotated with the mx_metadata note like `mxcli git commit` does. On a model-file conflict the base and theirs revisions are -checked out into temporary worktrees and `mx git-merge` runs (resolved for -the project's Mendix version): - -- what mx auto-resolves is staged and committed as the normal two-parent - merge commit, with the note attached; -- a conflict mx reports as `ManualResolutionRequired` is never merged - silently: the merge is aborted, and **both sides' changes are printed for - review** — at property level, from `mx diff` ("New type > Length: - 100 → 150" on ours, "→ 200" on theirs). Then decide: +checked out into temporary worktrees and the three project folders go to +`mx merge` (resolved for the project's Mendix version); the engine writes +its result into the working tree and owns the `.mpr`: + +- what the engine auto-resolves is staged, hash-checked and committed as the + normal two-parent merge commit, with the note attached. The engine's + `.mpr` carries both sides' document sets, so a unit one side added while + the other changed existing ones is kept, not dropped; +- a conflict the engine reports as `ManualResolutionRequired` (exit 2) is + never merged silently: the merge is aborted, and **both sides' changes are + printed for review** — at property level, from `mx diff` ("New type > + Length: 100 → 150" on ours, "→ 200" on theirs). Then decide: `--plan ` resolves **per unit** from the plan template written beside - the .mpr (`.merge-plan.json`): set `"take"` to `ours`/`theirs` per - unit, and the `.mpr` index is repaired from the mixed files (modified-on- - both units only; added/deleted/moved conflicts go to Studio Pro). - `--ours` or `--theirs` takes one side of every conflicted model file - (deterministic; mx is not consulted again). Resolutions that must mix both - sides *inside* one unit stay in Studio Pro (*Version Control › Merge - Changes Here*); + the .mpr (`.merge-plan.json`, which `mxcli git commit` never stages): + set `"take"` to `ours`/`theirs` per unit, and the `.mpr` index is repaired + from the mixed files (modified-on-both units only; added/deleted/moved + conflicts go to Studio Pro). `--ours` or `--theirs` takes one side of every + conflicted unit, hash-gated: when the sides' document sets differ (one side + added or deleted a unit), committing a whole side would orphan the other + side's documents, so the commit is refused and the merge aborted. Any other + engine exit is mx failing, not a decision: it is reported and the merge is + aborted cleanly. Resolutions that must mix both sides *inside* one unit + stay in Studio Pro (*Version Control › Merge Changes Here*); - a conflict in non-model files (text sources) is left in place for manual - resolution, like plain git. + resolution, like plain git: resolve them, `git add` them, and re-run the + command — it finishes the merge (model conflicts included) and commits. Merging is still never done by plain `git merge` on a Mendix repository: use `mxcli git merge` or Studio Pro, never a textual merge of `mprcontents/`. diff --git a/docs/plans/2026-10-30-git-merge-mx-folder-mode.md b/docs/plans/2026-10-30-git-merge-mx-folder-mode.md new file mode 100644 index 000000000..85de356f7 --- /dev/null +++ b/docs/plans/2026-10-30-git-merge-mx-folder-mode.md @@ -0,0 +1,57 @@ +# Rework `mxcli git merge` per ako's review (PR #1674) + +Decisions (user, 2026-10-30): switch the engine call to `mx merge` (folder mode); +keep `--ours`/`--theirs` but limited to conflicted units with a hash gate; one +push after the rework passes `make ci-local`. + +## Review claims → verified + +1. --ours/--theirs take App.mpr wholesale → orphan/missing documents. TRUE (strategy path, cmd_git_merge.go). +2. --plan takes ours' App.mpr → refuses on theirs' adds; error paths leave MERGE_HEAD. TRUE (applyPlan; abort only on stale). +3. Every non-zero mx exit treated as decision-needed. TRUE (code != 0 → plan + decision error; 129 = crash). +4. mx git-merge broken on Linux ("Git not found", 129). Reviewer-measured; repo finding corroborates mx merge works (0/2/129). +5. Should-fix: upstream merge conflicts (DONE, 2ef7e1068); merge-plan.json not junk-pathed; mixed-conflict advice; PrepareMxCommand nit. + +## Changes + +### A. Engine interface (cmd_git_merge.go) +- `mxMergeRunner func(baseDir, mineDir, theirsDir string, shas [3]string) (int, string, error)` — folders. +- realMxMerge: `mx merge ` + docker.PrepareMxCommand (11.10 LD_PRELOAD nit). +- baseDir/theirsDir: temp worktrees (unchanged); project dir inside = filepath.Join(dir, projDirRel). +- mineDir = the working project dir (filepath.Dir(abs mpr)). + +### B. Exit-code honesty +- 0 → stage model paths → fixHashes gate (MISSING/ORPHAN → refuse + git merge --abort) → commitTheMerge. +- 2 → property report (mx diff) + plan template (units = engine JSON's manual units; fallback git's conflicted ids) + git merge --abort + decision error. +- other (129…) → "mx merge failed (exit N)" + git merge --abort. Never a plan, never a decision suggestion. + +### C. --ours/--theirs limited +- No engine call (unchanged). Conflicted UNITS only from git's list (skip the .mpr). +- App.mpr takes the flag's side; then fixHashes gate: MISSING/ORPHAN → refuse + git merge --abort (loud, Studio Pro pointer). No silent drops. + +### D. --plan aborts everywhere +- invalid JSON / invalid take / not-modified-on-both / fixHashes refusal → git merge --abort before returning. +- stale-plan path already aborts (keep). + +### E. Re-entry +- At entry (non-plan): if MERGE_HEAD already exists, skip `git merge` and go straight to conflict classification. Supports: mixed conflict → resolve text files → `git add` → re-run command finishes the merge. + +### F. Mixed-conflict advice +- Early-return kept, reworded: resolve the text files, `git add` them, then re-run `mxcli git merge ` (it resumes, resolves the model, commits). + +### G. merge-plan.json is junk +- isJunkPath: `.merge-plan.json` never staged by `mxcli git commit`. + +### H. Words +- Long help, CHANGELOG git-merge entry, docs-site outside-studio-pro.md, teamserver-git SKILL.md "aborts the merge cleanly" — match the new reality. + +### I. Tests (gitmerge_test.go) +- Runner fixtures → folder form (assert dirs end in the project folder, contain App.mpr). +- exit 129 → no plan text, MERGE_HEAD gone, tree restored, error names mx. +- --theirs with ours-added unit → refuse + abort (no ORPHAN commit, MERGE_HEAD gone). +- --plan error paths → MERGE_HEAD gone. +- re-entry: staged text resolution + MERGE_HEAD → finishes. +- isJunkPath: x.merge-plan.json → junk. + +## Gate +- go vet ./cmd/mxcli/; focused go test; then make ci-local; then one push. From 4fa453f152e8c1d3bb44244de276e345da64bec7 Mon Sep 17 00:00:00 2001 From: engalar Date: Fri, 30 Oct 2026 12:24:07 +0800 Subject: [PATCH 44/44] docs(skills): regenerate the teamserver-git section index after the mx merge wording --- .claude/skills/mendix/teamserver-git/SKILL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/skills/mendix/teamserver-git/SKILL.md b/.claude/skills/mendix/teamserver-git/SKILL.md index 944bff868..9c4be9fe2 100644 --- a/.claude/skills/mendix/teamserver-git/SKILL.md +++ b/.claude/skills/mendix/teamserver-git/SKILL.md @@ -14,9 +14,9 @@ description: "Commit, branch and push a Mendix project in git the way Studio Pro | 42-55 | Branches: create them on the server | | 56-97 | Commit from the agent | | 98-112 | Push the branch and the notes together, and soon | -| 113-132 | Never rewrite or merge model history with git | -| 133-143 | Reading what Studio Pro recorded | -| 144-152 | Checklist | +| 113-134 | Never rewrite or merge model history with git | +| 135-145 | Reading what Studio Pro recorded | +| 146-154 | Checklist | # Working with Git on a Mendix Team Server Project