Skip to content

test(parity): compare with the reference's own functions and runs, and close what they found - #41

Merged
aledbf merged 17 commits into
mainfrom
test/differential-runtime
Oct 26, 2026
Merged

aledbf merged 17 commits into
mainfrom
test/differential-runtime

Conversation

@aledbf

@aledbf aledbf commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The parity matrix, unit tests and e2e suites stayed green while the CLI still differed from the reference in ways nobody had written a case for. This PR adds two kinds of tests that take expected values from the reference itself rather than from hand-written cases, and fixes everything they found.

  • TS oracle goldens (tools/tsoracle, internal/tsoracle). The oracle runs the reference's own functions from reference/src, bundled with the reference's esbuild, over JSON cases. task testdata:ts-oracle records the results in internal/*/testdata/ts-oracle, and the Go ports are compared against them. A case marked divergence has to keep differing, so fixing it means deleting the mark. About 480 cases cover the Dockerfile helpers, mergeConfiguration and the metadata label, the Feature build helpers, variable substitution and devcontainerIdForLabels, getRef, the compose helpers, and more.
  • Differential runtime tests (TestDifferentialUp, TestDifferentialBuild, next to TestDifferentialReadConfiguration). Each one runs up or build on every upstream fixture with both CLIs and compares what a user sees: the JSON result, the container and image configuration (with devcontainer.metadata parsed), mounts, and id inside the container. The known-differences list can only shrink. Ten entries remain, all the accepted legacy GitHub Release Feature limitation (5 fixtures × up/build).

Bugs fixed

Each fix comes with a golden case, a unit test or a parity case that fails before the fix.

  • ${devcontainerId} hashed labels with &<> and U+2028/U+2029 escaped and keys sorted by bytes, so some label sets produced a different id.
  • A stage name lost its case (AS Final), CRLF Dockerfiles got AS after the \r, and a Dev target was not found. Nested defaults like ${IMG:-${REG}/app}, the unresolved-variable error, ${...Basename} of /, and win32 trailing separators all differed too.
  • devcontainers-contrib refs were not redirected to devcontainers-extra. The features label escaped && and <>. readComposeVersionPrefix kept \r. parseMount cut source=/a=b in the wrong place. mergeConfiguration took runArgs from metadata entries, where TS takes them from the config only.
  • The metadata label is where and as the reference puts it. It holds the raw entries (base image, Features, the config's picked properties, unsubstituted). It goes on the container only for an image with no Features, and on the image otherwise. It is written as JSON.stringify writes it. Containers are read back the way getImageMetadataFromContainer reads them.
  • Compose build goes through docker compose build with the label override, so the image carries compose's labels and the project name follows getProjectName. --secrets-file does not apply there (documented).
  • up's Features image now gets the config's containerEnv as ENV lines, as build already did.
  • Base images resolve with the automatic platform ARGs, so FROM img-$TARGETARCH names the right image. EnsureFinalStageName now returns TS's errors for a Dockerfile with no FROM, or with an unparseable final FROM.
  • exec probes the environment as the configured remoteUser. Before, it probed as the metadata user and gave the command that user's HOME and PATH. New runtime parity case: exec.config-remote-user-probe-success.
  • read-configuration exits 1 when --container-id cannot be inspected. New parity case: read-configuration.container-id-missing-error.
  • compose config errors and legacy Feature errors are ContainerErrors, and a failed array lifecycle command is named as TS names it.

Deliberate divergences (docs/DIVERGENCES.md, each recorded as a golden divergence case)

  • A final FROM line continued with a backslash keeps its stage name. TS reads only the physical line and writes FROM \ AS dev, a Dockerfile no builder accepts. Here BuildKit reads the stage's name, and every other FROM line is handled exactly as TS handles it.

  • Dockerfile ARG/ENV values are read with BuildKit's parser. Upstream fixes #1213 (colons in ${localEnv:VAR:default}) and #881 (:ro in compose volumes) were already ported, and both are newer than the pinned reference. Object keys inside metadata label values are sorted.

  • Compose builds may read the Features Dockerfile outside the build context. Now that a compose build goes through docker compose build, bake on buildx ≥ 0.38 refused the Features Dockerfile, written in the cache folder, unless --allow=fs.read was passed, which compose cannot do. TestE2E_ComposeBuildLabel failed this way on the runners. The reference fails there too. This CLI starts its compose processes with BUILDX_BAKE_ENTITLEMENTS_FS=0 unless that variable is already set.

A # escape= directive now follows TS (recorded in the golden).

CI gate for the goldens

testdata:ts-oracle:check was not run by any CI job, so a stale golden would never be caught. TestTSOracleGoldensMatchTheReference compares every golden with what the reference returns now, and task parity:contract, which CI runs after task reference, now runs it too. It caught a hand-edited golden in this branch.

Validation

Check Result
go vet ./... on every commit pass
go test -race ./... (Docker unreachable) pass
task lint pass
task parity:contract (incl. the new oracle gate) pass
TestDifferentialReadConfiguration pass
TestDifferentialUp / TestDifferentialBuild (Docker, local) pass (up 15 min, build 11 min); skipped: the 5 known legacy-Feature fixtures and 4 arm64 fixtures
go tool mutate over this diff, Docker unreachable, -j 1 547 refused, 0 survived, 53 did not build

go tool mutate -j 4 reported false survivors in extendImageWithFeatures, a different set on each run. Every one of them was refused when run serially, or when the mutation was applied by hand. Mutants that write into the package directory interfere with each other's runs; this is a go-tools follow-up, and it changes nothing here.

🤖 Generated with Claude Code

@aledbf
aledbf force-pushed the test/differential-runtime branch from 4d8c0eb to 619986b Compare October 25, 2026 23:50
Base automatically changed from test/mutation-nodocker to main October 26, 2026 00:09
aledbf and others added 17 commits October 25, 2026 21:09
Tests that claimed TS behaviour carried expected values typed by hand,
so a typo or a reference upgrade went unseen, and tests written against
Go's behaviour missed real divergences. tools/tsoracle runs the real TS
functions from reference/src over JSON cases; `task testdata:ts-oracle`
records their results in internal/*/testdata/ts-oracle and
`testdata:ts-oracle:check` fails on a stale one. 483 cases cover
parseVersion, isEarlierVersion, the Dockerfile helpers, mergeConfiguration
and the metadata label, the feature build helpers, variable substitution
and devcontainerIdForLabels, getRef and getCollectionRef, the compose
helpers, escapeQuotesForShell and qualifyImageName. A case marked as a
divergence must keep differing, so a fix removes its mark.

They found, and this fixes:
- ${devcontainerId} escaped &, < and > and U+2028/U+2029 and sorted keys
  by bytes, so some id labels hashed to another id;
- a stage name lost its case (`AS Final`), CRLF Dockerfiles got `AS`
  after the '\r', and a target `Dev` was not found;
- ${IMG:-${REG}/app} and other nested defaults resolved differently;
- varsub's unresolved variable error, ${...Basename} of '/' and win32
  trailing separators;
- devcontainers-contrib refs were not redirected to devcontainers-extra;
- the features label escaped && and <>;
- readComposeVersionPrefix kept '\r', parseMount cut `source=/a=b`
  wrong, and `external=false` is external as in TS;
- mergeConfiguration took runArgs from metadata entries; TS takes them
  from the config only.

docs/DIVERGENCES.md records the BuildKit Dockerfile parsing and the
upstream #1213 and #881 ports, which stay as they are.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TestDifferentialReadConfiguration compares read-configuration over every
upstream fixture; up and build had only the hand-written parity cases.
TestDifferentialUp and TestDifferentialBuild run each command with the
TS oracle and with this CLI, one after the other on copies of the
fixture, and compare what a user of the result sees: the command's JSON,
the container's and the image's configuration (user, env, entrypoint,
command, labels with devcontainer.metadata parsed, mounts, init,
privileged, capabilities, security options) and the remote user's id.
Paths, hashes, digests, times and the devcontainerId are normalized.

The first run found 20 of 36 fixtures differing on up and 19 on build;
differentialRuntimeKnown lists each with its reason and can only shrink.
`task parity:differential` runs the three comparisons.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS runs an array-form lifecycle command without a shell and its failure
message is "Command failed: " with the arguments joined by spaces. Go
reported the sh -c form it builds to run the array, so the up error
message differed from the reference (fixture poetry-example).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…rence

TS throws a ContainerError with its own description for a failing
`docker compose config` ("An error occurred retrieving the Docker
Compose configuration.", the message naming the command) and for a
legacy Feature id (the error text as description and message), and up
and build report a ContainerError as it is. Go wrapped both into its
generic "An error occurred setting up/building the container: ..."
description.

compose config now runs as TS does, `--profile * config` with the YAML
output read, so services behind a profile are found and the failed
command line in the message is the reference's. up and build report any
ContainerError from deeper layers unchanged, which replaces the
prefix-matching of the legacy Feature message.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS findContainerAndIdLabels inspects the --container-id with `docker
inspect`, whose failure throws, so read-configuration exits 1 with no
result. Go ignored the inspect error and printed the configuration as
if there were no container. The failure is now logged and exits 1; the
runtime parity case read-configuration.container-id-missing-error
compares both CLIs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS exec merges the configuration's remoteUser into the container's
metadata and creates the container properties, the environment probe
included, with mergedConfig.remoteUser. Go probed as the metadata or
container user and switched to the configuration's remoteUser only to
run the command, so the command got the probing user's HOME and PATH.
The runtime parity case exec.config-remote-user-probe-success compares
both CLIs on a container whose config gains a remoteUser after up.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS findBaseImage resolves FROM with the automatic BUILD*/TARGET* ARGs
(imageMetadata.ts globalBuildxPlatformArgs); up, build and the compose
build path passed none, so `FROM img-$TARGETARCH` named another image
and its metadata and user were lost. The target platform is --platform,
else the host, as TS devContainers.ts targetPlatformInfo.

EnsureFinalStageName now returns TS's errors for a Dockerfile without a
FROM line or with an unparseable final FROM, where it used to carry on
with the default stage name; build, up and the compose path report it.
The two ts-oracle divergences for those cases are gone, and new golden
cases cover platform ARGs in FROM and an unparseable final FROM.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS getFeaturesBuildOptions adds devcontainer.json's containerEnv to the
Features Dockerfile as ENV lines (#{containerEnvMetadata}) for every
command; build passed it, up did not, so an image built by up lacked
those variables (fixture image-metadata-containerEnv, whose up entry
now names only the label difference left).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS writes devcontainer.metadata with JSON.stringify: &, < and > as they
are, and each entry's properties in the order its pick produced them
(id and pickFeatureProperties for a Feature, pickConfigProperties for
the configuration). Go escaped &<> and wrote Entry's field order.

Object values keep sorted keys, a decoded Go map having no order; that
remaining byte-level difference is recorded in docs/DIVERGENCES.md and
in the ts-oracle goldens, which now also hold pick-ordered cases that
match.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS records devcontainer.metadata as getDevcontainerMetadata's raw
entries: the base image's, the Features' and pick(config.raw,
pickConfigProperties), variables unresolved and runArgs left out. Only
a container from an image with no Features gets the label (JSON of the
entries, "[]" when none); a Dockerfile or Features build, and every
image `build` makes, puts it on the image with the configuration's
entry. Go always labelled the container (repeating the config entry
for Dockerfile builds), left the config entry out of up's image label
and the base image's entries out of build's, wrote substituted values
and runArgs, and wrote "" for no entries.

Since a CLI-created container's metadata now records the configuration,
up, exec, run-user-commands and read-configuration read it as TS
getImageMetadataFromContainer does: the label's entries with their
variables substituted, then only the updateable properties (remoteUser,
userEnvProbe, remoteEnv) for a container with every id label, else the
configuration's entry. The merged configuration keeps the substituted
entries.

build of a compose configuration now goes through the compose port of
buildAndExtendDockerCompose, as TS does: `docker compose build` with
the label build override, also for an image-only service without
Features, so the image carries compose's project and service labels and
the project name follows TS getProjectName. --secrets-file does not
apply there (documented); --label keeps applying through a FROM build.

Every differentialRuntimeKnown entry about labels is gone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Cover what no test refused: read-configuration fails through one
helper whose exit code the existing failure tests hold, and the missing
--container-id test checks it is the inspect failure that ends the
command; prepareDockerfileBuild's target and the cleanup of the named
Dockerfile; a parallel lifecycle command's header only on success; the
52-digit padding of the dev container id; a digest with a second colon;
an empty request to the TS oracle. The final-stage label condition loses
a clause that could never differ, and up's container metadata reader
names the runtime lanes that run it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Mutation testing found that no test held node path.win32.basename's
drive rule (only a letter and a colon are a drive) in winBasename. The
52-digit check of ComputeDevContainerID gets a mutate-exempt: < 53
pads nothing more.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Mutation testing found that no test held the stage-by-name lookup
behind StageLabels, which keeps a user's metadata label of the targeted
stage.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS ensureDockerfileHasFinalStageName reads only the physical last FROM
line, so `FROM \` followed by `alpine AS base` becomes `FROM \ AS dev`,
a Dockerfile no builder accepts. Ask BuildKit for that stage's name
instead; every other FROM line keeps TS's handling. The ts-oracle golden
records TS's answer as a divergence case, and docs/DIVERGENCES.md
explains it. A `# escape=` directive now follows TS too, also recorded
in the golden.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
mutate without Docker over this branch's diff left edits no test
refused. A win32 basename keeps `{:foo` whole, as node's
path.win32.basename does ('{' lowercases past 'z'); oracletest names
the JSON error of a bad recorded or Go result; a case's subtest name
takes the first of its name and fn#index. Dockerfile drops the escape
token nothing reads since base images resolve as TS does, finalStageName
leans on instructions.Parse returning no stages with its error, and the
two index edits that cannot change a match are exempt. stageLabel loses
a check that could not fail: on every FROM line BuildKit accepts, its
continuations joined, the FROM-line expression reads BuildKit's label.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
testdata:ts-oracle:check ran in no CI job, so a reference upgrade or a
hand-edited golden went unseen. TestTSOracleGoldensMatchTheReference
bundles the oracle and compares every golden with what the reference's
functions return; task parity:contract, which CI runs after task
reference, now runs it, and testdata:ts-oracle:check is that test.
tsOracle keeps only the bundle and the node process, the lane test
holding it; refreshTSOracleFiles reads, compares and writes, tested
with a fake oracle. build.mjs resolves a symlinked reference/, since
esbuild reports real paths.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Building a compose configuration now goes through `docker compose
build`, as in TS. Compose builds through bake, and bake on buildx 0.38
and later refuses the Features Dockerfile, written under the cache
folder outside the build context, unless --allow=fs.read is passed,
which compose cannot do. TestE2E_ComposeBuildLabel failed that way on
the runners. The compose processes this CLI starts get
BUILDX_BAKE_ENTITLEMENTS_FS=0 unless the variable is already set. The
reference fails there; docs/DIVERGENCES.md records it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@aledbf
aledbf force-pushed the test/differential-runtime branch from 619986b to 81a93af Compare October 26, 2026 00:09
@aledbf
aledbf merged commit e63967b into main Oct 26, 2026
16 of 24 checks passed
@aledbf
aledbf deleted the test/differential-runtime branch October 26, 2026 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant