docs(agent): one fixture layout block for every authoring recipe

openspec/changes/unified-rule-fixtures/tasks.md → openspec/changes/archive/2026-10-26-unified-rule-fixtures/tasks.md

@@ -35,6 +35,5 @@ Stacked, merging forward with `gh stack merge`. The task groups describe the wor
 - [x] 4.1 Write the shared fixture section and point `create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, `improve-rule` and `validate-rule` at it, removing their per-engine layout copies. Pass explicit paths to prettier, never `src/` (memory: prompt parity). Verify the prompt parity tests and `pnpm lint`.
 - [x] 4.2 `reference.json`: a case list for every engine, `grouping: case-directories`, and an incremented corpus `version`. Verify `reference.test.ts` covers every scenario in the `cli-conformance-reference` delta.
 - [x] 4.3 Update `.taskless/rules` READMEs, CONTRIBUTING and any doc naming `.tests/`. Verify `grep -rn '\.tests/'` outside migrations, the delivery JIT conversion and the CHANGELOG returns nothing unexpected.
-- [ ] 4.4 Note the new layout and corpus version in the CLI collab artifact as information for Cloud, stating that no action is required because delivery migrates the old format just in time (memory: cloud collab artifacts). Verify the artifact reads back with the note.
+- [x] 4.4 Tell Cloud about the new layout and corpus version. Filed as taskless/taskless#398 instead of the collab artifact: delivery migrates the old format just in time, so nothing breaks when the CLI ships, and Cloud follows by keying the new format on CLI >= 0.13.4 and validating against the nightly.
 - [x] 4.5 Run `pnpm typecheck` and `pnpm lint` across the stack tip. Archive the change on the tip (`pnpm openspec archive unified-rule-fixtures -y`), confirming every prior scenario survives per AGENTS.md.
-- [ ] 4.6 Land the stack with `gh stack merge`, after review.

openspec/specs/cli-agent-authoring/spec.md

@@ -28,7 +28,7 @@ The `create-vale-rule` recipe SHALL instruct the agent to produce three artifact
 
 1. A Vale style file at `.taskless/rules/vale/<id>/<id>.yml`.
 2. That rule's own `.taskless/rules/vale/<id>/.vale.ini`, declaring the matchers that scope it and enabling it as `<id>.<id> = YES`.
-3. `pass/` and `fail/` fixture documents under `.taskless/rules/vale/<id>/.tests/`.
+3. Fixture case directories under `.taskless/rules/vale/<id>/tests/pass/` and `tests/fail/`, laid out as the shared fixture section describes.
 
 The recipe SHALL state that scope is declared in the rule's own config, that no shared file is edited, and that the project-wide config is assembled rather than authored.
 
@@ -241,3 +241,28 @@ Enforcing this is the recipe's job. `verify` SHALL NOT lint fixtures and SHALL N
 
 - **WHEN** `verify` runs a rule whose `pass` fixture would fail the repository's linter
 - **THEN** `verify` SHALL report on the rule's behavior alone, and SHALL NOT fail the rule over that fixture's style
+
+### Requirement: Authoring recipes share one fixture section
+
+Every recipe in which the agent writes or extends fixtures (`create-sg-rule`, `create-vale-rule`, `create-runtime-rule`, `improve-rule (anonymous)` and `validate-rule`) SHALL direct the agent to one shared fixture section and SHALL NOT carry its own copy of the layout. The shared section SHALL state:
+
+1. The layout: `tests/pass/<case>/` and `tests/fail/<case>/`, with the case directory named for what the case shows.
+2. That a case is judged by the engine's own check with the case directory as the root, so a fixture for a scoped rule places its files at a path the scope accepts. For example, an ast-grep rule scoped to `.github/workflows/*.yml` needs `tests/fail/<case>/.github/workflows/ci.yml`.
+3. That a `fail/` case needs one finding and a `pass/` case none, and that nothing about position, message, or count is asserted.
+4. What a "scope excluded this case" failure means, and that the fix is the fixture's path or the scope, not the pattern.
+5. That every case must be read: a case the rule read no file in fails in either bucket, the project's `.gitignore` applies inside a case, and a case with no file is refused.
+
+A per-engine recipe SHALL keep only what is that engine's own, such as Vale's scope reach and the fact that a runtime case is the check's root.
+
+**Rationale.** The fixture contract is now one contract. Three copies of it would drift the way the three fixture readers did, and the path rule for scoped rules is the one new thing authors have to learn. It belongs where every recipe reaches it.
+
+#### Scenario: Every authoring recipe reaches the same fixture guidance
+
+- **WHEN** the agent follows any recipe that writes fixtures
+- **THEN** the recipe SHALL direct it to the shared fixture section
+- **AND** SHALL NOT restate the layout in its own words
+
+#### Scenario: A scoped rule's fixture path is taught
+
+- **WHEN** the agent writes fixtures for a rule that declares `files:` or Vale matchers
+- **THEN** the shared section SHALL direct it to place each case's files at a path the scope accepts

openspec/specs/cli-conformance-reference/spec.md

@@ -33,52 +33,6 @@ behaving correctly, so the bump is the only signal the corpus owes it.
 - **THEN** the files a generator must produce SHALL be carried apart from the cases both sides' rules must satisfy
 - **AND** no file SHALL appear in both
 
-### Requirement: The corpus states how its fixtures group into cases
-
-Each corpus entry's `tests` SHALL carry a `grouping` field naming how its fixture
-files group into cases, the flat list of fixture files with their contents, and
-— where the grouping is the CLI's own — an explicit list of cases.
-
-A consumer SHALL be able to recover which files belong to which case by reading
-published fields alone. Deriving it from path prefixes requires knowing that a
-`runtime` case is a directory and a `vale` case is a document, which is a fact
-about the CLI's layout that every consumer would otherwise transcribe, and a
-transcribed fact goes stale silently when the layout changes.
-
-Each case SHALL name the `bucket` it asserts (`pass` or `fail`), its own `name`,
-the `path` the engine is pointed at, and the fixture files it holds. Case files
-SHALL be named by path rather than by repeated content, and every such path
-SHALL resolve to an entry in the same `tests.files` list.
-
-`bucket` SHALL be spelled `pass`/`fail`, matching the directories the CLI reads,
-rather than an individual engine's vocabulary.
-
-#### Scenario: A runtime case is a directory of files
-
-- **WHEN** a consumer reads the `runtime` entry
-- **THEN** `grouping` SHALL be `case-directories`
-- **AND** each case SHALL name the directory the check is handed as its root
-- **AND** a case whose evidence spans two files SHALL list both
-
-#### Scenario: A vale case is one document
-
-- **WHEN** a consumer reads the `vale` entry
-- **THEN** `grouping` SHALL be `case-documents`
-- **AND** each case SHALL name the document and hold exactly that one file
-
-#### Scenario: An ast-grep entry says its grouping is not the CLI's
-
-- **WHEN** a consumer reads the `sg` entry
-- **THEN** `grouping` SHALL be `ast-grep-test`
-- **AND** the ast-grep test file SHALL be carried whole
-- **AND** no case list SHALL be published, because the grouping inside that file belongs to ast-grep's own documented schema rather than to the CLI
-
-#### Scenario: Every case file resolves
-
-- **WHEN** the corpus is generated
-- **THEN** every path a case names SHALL appear in that entry's `tests.files`
-- **AND** every file in `tests.files` SHALL belong to exactly one case, where a case list is published
-
 ### Requirement: The corpus states where a rule lives
 
 The corpus SHALL publish the rule layout: the root every rule hangs off, the
@@ -130,3 +84,52 @@ disagreement about the subject the rule addresses.
 - **WHEN** the commands an `enforcedBy` value names are renamed, so an existing value would mean a different command
 - **THEN** the corpus `version` SHALL be incremented
 - **AND** a consumer reading the new version SHALL find only the current command names among the `enforcedBy` values
+
+### Requirement: The corpus publishes every engine's fixtures as case directories
+
+Each corpus entry's `tests` SHALL carry a `grouping` field naming how its fixture
+files group into cases, the flat list of fixture files with their contents, and
+an explicit list of cases.
+
+A consumer SHALL be able to recover which files belong to which case by reading
+published fields alone, so that the case layout stays a published fact and is
+not transcribed by every consumer. A transcribed fact goes stale silently when
+the layout changes, and this change is such a change.
+
+Every engine SHALL publish `grouping` as `case-directories` with an explicit case
+list. The field stays even though it now has one value: a consumer asserts on
+it, and it is what a later layout change would move. The `case-documents` and
+`ast-grep-test` values SHALL NOT be published, because no engine uses them.
+
+Each case SHALL name the `bucket` it asserts (`pass` or `fail`), its own `name`,
+the `path` the engine is pointed at, and the fixture files it holds. Case files
+SHALL be named by path rather than by repeated content, and every such path
+SHALL resolve to an entry in the same `tests.files` list.
+
+`bucket` SHALL be spelled `pass`/`fail`, matching the directories the CLI reads,
+rather than an individual engine's vocabulary.
+
+#### Scenario: Every engine publishes case directories
+
+- **WHEN** a consumer reads the `sg`, `vale`, or `runtime` entry
+- **THEN** `grouping` SHALL be `case-directories`
+- **AND** each case SHALL name the directory the engine is handed as its root
+- **AND** a case whose evidence spans two files SHALL list both
+
+#### Scenario: No engine-native test file is published
+
+- **WHEN** a consumer reads the `sg` entry
+- **THEN** its `tests.files` SHALL hold no ast-grep test YAML
+- **AND** every file SHALL belong to a published case
+
+#### Scenario: A single-document case is still a directory
+
+- **WHEN** a consumer reads the `vale` entry and a case holds a single document
+- **THEN** the case SHALL name its directory as its `path`
+- **AND** SHALL list that one document as its file
+
+#### Scenario: Every case file resolves
+
+- **WHEN** the corpus is generated
+- **THEN** every path a case names SHALL appear in that entry's `tests.files`
+- **AND** every file in `tests.files` SHALL belong to exactly one case

openspec/specs/cli-generated-rule-delivery/spec.md

@@ -81,11 +81,15 @@ When the CLI writes a runtime rule from a response whose `entitlement.runtimeSig
 ### Requirement: A delivered file set is verified against its signatures
 
 Before writing any file of a served rule, the CLI SHALL verify that every signature names a
-delivered file, that every delivered file outside `.tests/` has exactly one signature, and that
+delivered file, that every delivered file outside `tests/` has exactly one signature, and that
 the algoVersion-1 `canonicalHash` of each such file's content equals its signature. For a
 runtime rule it SHALL also verify that the singular `signature` equals the `check.ts` entry.
 Any failure SHALL refuse the whole rule and write nothing.
 
+A served file under the legacy `.tests/` directory SHALL be treated as a fixture for this check,
+like a file under `tests/`. A service that has not yet moved to the new layout still serves its
+fixtures there, unsigned, and refusing them as unsigned files would refuse every rule it serves.
+
 #### Scenario: A corrupted file is refused
 
 - **WHEN** a served file's content does not hash to its entry in `signatures`
@@ -94,32 +98,79 @@ Any failure SHALL refuse the whole rule and write nothing.
 
 #### Scenario: An unsigned file is refused
 
-- **WHEN** a served file outside `.tests/` has no entry in `signatures`
+- **WHEN** a served file outside `tests/` has no entry in `signatures`
 - **THEN** the CLI SHALL refuse the rule
 
 #### Scenario: Fixtures need no signature
 
-- **WHEN** a served rule carries files under `.tests/` that have no entry in `signatures`
+- **WHEN** a served rule carries files under `tests/` that have no entry in `signatures`
 - **THEN** the CLI SHALL accept them
 
+#### Scenario: Legacy fixtures need no signature either
+
+- **WHEN** a served rule carries files under `.tests/` that have no entry in `signatures`
+- **THEN** the CLI SHALL accept them as fixtures
+
 ### Requirement: A delivered rule replaces its directory
 
 After verification, the CLI SHALL write a served rule so that its directory holds exactly the
 served files: every file in the directory that the served set does not contain SHALL be removed,
-including under `.tests/`, since a served set always carries the rule's fixtures. The CLI SHALL
+including under `tests/`, since a served set always carries the rule's fixtures. The CLI SHALL
 create each served file's parent directories as needed, so a nested path such as
-`captures/env.yml` or `.tests/fail/case.ts` is written wherever it lands. When a stale file cannot be removed, the CLI SHALL say that the served
+`captures/env.yml` or `tests/fail/eval-call/case.ts` is written wherever it lands. When a stale file cannot be removed, the CLI SHALL say that the served
 bytes were written and name each entry it could not remove.
 
+A served set whose fixtures are in the old format, under `.tests/`, SHALL be migrated just in
+time: converted the same way migration 12 converts a local rule, written in the `tests/`
+layout, and never written under `.tests/`. Fixtures carry no signature, so the conversion
+changes no signed byte, and the CLI never writes a layout its own `verify` does not read. This
+is a permanent reading of the old format, not a transition step. The CLI SHALL NOT depend on
+the service emitting the new layout, and a served set already in the new layout SHALL be
+written unchanged. A rule served with several legacy test files SHALL keep every snippet:
+case names SHALL NOT collide across them, nor with a case the served set already carries
+under `tests/`, nor with the name a fixture moved as served keeps. `sg test`'s
+`__snapshots__/` SHALL be dropped and named. Fixtures the converter cannot place SHALL be written
+under `tests/` at their served path below `.tests/`, and the delivery SHALL say so, naming them.
+`verify` then refuses them by path, so an unplaced fixture is never mistaken for a passing one.
+
 #### Scenario: A local extra file does not survive
 
 - **WHEN** a rule directory holds `captures/extra.yml` and the served set does not
 - **THEN** after the write `captures/extra.yml` SHALL NOT exist
 
 #### Scenario: Nested paths are written
 
-- **WHEN** a served set carries `.tests/fail/case.ts` and the rule directory does not exist
-- **THEN** the CLI SHALL create `.tests/fail/` and write the file
+- **WHEN** a served set carries `tests/fail/eval-call/case.ts` and the rule directory does not exist
+- **THEN** the CLI SHALL create `tests/fail/eval-call/` and write the file
+
+#### Scenario: Legacy fixture paths are written in the new layout
+
+- **WHEN** a served `sg` rule carries its fixtures as `.tests/<id>-test.yml` with `valid:` and `invalid:` snippets
+- **THEN** the CLI SHALL write them as case directories under `tests/pass/` and `tests/fail/`
+- **AND** the rule directory SHALL NOT contain `.tests/` afterwards
+
+#### Scenario: Several legacy test files keep every snippet
+
+- **WHEN** a served `sg` rule carries two legacy test files, each with one `invalid:` snippet
+- **THEN** the CLI SHALL write two `fail/` cases, one per snippet
+
+#### Scenario: A served set in both layouts keeps its cases apart
+
+- **WHEN** a served `sg` rule carries `tests/fail/fail-01/` and a legacy test file with one `invalid:` snippet
+- **THEN** the CLI SHALL write the converted snippet as `tests/fail/fail-02/`
+- **AND** `tests/fail/fail-01/` SHALL hold only what was served there
+
+#### Scenario: A fixture the converter cannot place is written and named
+
+- **WHEN** a served `sg` rule's `files:` and `ignores:` admit no path the converter can derive for its `.tests/<id>-test.yml`
+- **THEN** the CLI SHALL write that file at `tests/<id>-test.yml`, not under `.tests/`
+- **AND** the delivery SHALL warn, naming the file and why
+- **AND** the next `verify` of the rule SHALL report `tests/<id>-test.yml` as a stray entry
+
+#### Scenario: A served set in the new layout is written as served
+
+- **WHEN** a served set carries its fixtures under `tests/pass/<case>/` and `tests/fail/<case>/`
+- **THEN** the CLI SHALL write them at those paths without conversion
 
 #### Scenario: A generated rule's revision is confirmed