Repository navigation
test(parity): compare with the reference's own functions and runs, and close what they found - #41
Merged
Merged
Conversation
aledbf
force-pushed
the
test/differential-runtime
branch
from
October 25, 2026 23:50
4d8c0eb to
619986b
Compare
Tests that claimed TS behaviour carried expected values typed by hand,
so a typo or a reference upgrade went unseen, and tests written against
Go's behaviour missed real divergences. tools/tsoracle runs the real TS
functions from reference/src over JSON cases; `task testdata:ts-oracle`
records their results in internal/*/testdata/ts-oracle and
`testdata:ts-oracle:check` fails on a stale one. 483 cases cover
parseVersion, isEarlierVersion, the Dockerfile helpers, mergeConfiguration
and the metadata label, the feature build helpers, variable substitution
and devcontainerIdForLabels, getRef and getCollectionRef, the compose
helpers, escapeQuotesForShell and qualifyImageName. A case marked as a
divergence must keep differing, so a fix removes its mark.
They found, and this fixes:
- ${devcontainerId} escaped &, < and > and U+2028/U+2029 and sorted keys
by bytes, so some id labels hashed to another id;
- a stage name lost its case (`AS Final`), CRLF Dockerfiles got `AS`
after the '\r', and a target `Dev` was not found;
- ${IMG:-${REG}/app} and other nested defaults resolved differently;
- varsub's unresolved variable error, ${...Basename} of '/' and win32
trailing separators;
- devcontainers-contrib refs were not redirected to devcontainers-extra;
- the features label escaped && and <>;
- readComposeVersionPrefix kept '\r', parseMount cut `source=/a=b`
wrong, and `external=false` is external as in TS;
- mergeConfiguration took runArgs from metadata entries; TS takes them
from the config only.
docs/DIVERGENCES.md records the BuildKit Dockerfile parsing and the
upstream #1213 and #881 ports, which stay as they are.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
TestDifferentialReadConfiguration compares read-configuration over every upstream fixture; up and build had only the hand-written parity cases. TestDifferentialUp and TestDifferentialBuild run each command with the TS oracle and with this CLI, one after the other on copies of the fixture, and compare what a user of the result sees: the command's JSON, the container's and the image's configuration (user, env, entrypoint, command, labels with devcontainer.metadata parsed, mounts, init, privileged, capabilities, security options) and the remote user's id. Paths, hashes, digests, times and the devcontainerId are normalized. The first run found 20 of 36 fixtures differing on up and 19 on build; differentialRuntimeKnown lists each with its reason and can only shrink. `task parity:differential` runs the three comparisons. Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS runs an array-form lifecycle command without a shell and its failure message is "Command failed: " with the arguments joined by spaces. Go reported the sh -c form it builds to run the array, so the up error message differed from the reference (fixture poetry-example). Co-Authored-By: Claude Opus 5.5 <[email protected]>
…rence
TS throws a ContainerError with its own description for a failing
`docker compose config` ("An error occurred retrieving the Docker
Compose configuration.", the message naming the command) and for a
legacy Feature id (the error text as description and message), and up
and build report a ContainerError as it is. Go wrapped both into its
generic "An error occurred setting up/building the container: ..."
description.
compose config now runs as TS does, `--profile * config` with the YAML
output read, so services behind a profile are found and the failed
command line in the message is the reference's. up and build report any
ContainerError from deeper layers unchanged, which replaces the
prefix-matching of the legacy Feature message.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS findContainerAndIdLabels inspects the --container-id with `docker inspect`, whose failure throws, so read-configuration exits 1 with no result. Go ignored the inspect error and printed the configuration as if there were no container. The failure is now logged and exits 1; the runtime parity case read-configuration.container-id-missing-error compares both CLIs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS exec merges the configuration's remoteUser into the container's metadata and creates the container properties, the environment probe included, with mergedConfig.remoteUser. Go probed as the metadata or container user and switched to the configuration's remoteUser only to run the command, so the command got the probing user's HOME and PATH. The runtime parity case exec.config-remote-user-probe-success compares both CLIs on a container whose config gains a remoteUser after up. Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS findBaseImage resolves FROM with the automatic BUILD*/TARGET* ARGs (imageMetadata.ts globalBuildxPlatformArgs); up, build and the compose build path passed none, so `FROM img-$TARGETARCH` named another image and its metadata and user were lost. The target platform is --platform, else the host, as TS devContainers.ts targetPlatformInfo. EnsureFinalStageName now returns TS's errors for a Dockerfile without a FROM line or with an unparseable final FROM, where it used to carry on with the default stage name; build, up and the compose path report it. The two ts-oracle divergences for those cases are gone, and new golden cases cover platform ARGs in FROM and an unparseable final FROM. Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS getFeaturesBuildOptions adds devcontainer.json's containerEnv to the
Features Dockerfile as ENV lines (#{containerEnvMetadata}) for every
command; build passed it, up did not, so an image built by up lacked
those variables (fixture image-metadata-containerEnv, whose up entry
now names only the label difference left).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS writes devcontainer.metadata with JSON.stringify: &, < and > as they are, and each entry's properties in the order its pick produced them (id and pickFeatureProperties for a Feature, pickConfigProperties for the configuration). Go escaped &<> and wrote Entry's field order. Object values keep sorted keys, a decoded Go map having no order; that remaining byte-level difference is recorded in docs/DIVERGENCES.md and in the ts-oracle goldens, which now also hold pick-ordered cases that match. Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS records devcontainer.metadata as getDevcontainerMetadata's raw entries: the base image's, the Features' and pick(config.raw, pickConfigProperties), variables unresolved and runArgs left out. Only a container from an image with no Features gets the label (JSON of the entries, "[]" when none); a Dockerfile or Features build, and every image `build` makes, puts it on the image with the configuration's entry. Go always labelled the container (repeating the config entry for Dockerfile builds), left the config entry out of up's image label and the base image's entries out of build's, wrote substituted values and runArgs, and wrote "" for no entries. Since a CLI-created container's metadata now records the configuration, up, exec, run-user-commands and read-configuration read it as TS getImageMetadataFromContainer does: the label's entries with their variables substituted, then only the updateable properties (remoteUser, userEnvProbe, remoteEnv) for a container with every id label, else the configuration's entry. The merged configuration keeps the substituted entries. build of a compose configuration now goes through the compose port of buildAndExtendDockerCompose, as TS does: `docker compose build` with the label build override, also for an image-only service without Features, so the image carries compose's project and service labels and the project name follows TS getProjectName. --secrets-file does not apply there (documented); --label keeps applying through a FROM build. Every differentialRuntimeKnown entry about labels is gone. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Cover what no test refused: read-configuration fails through one helper whose exit code the existing failure tests hold, and the missing --container-id test checks it is the inspect failure that ends the command; prepareDockerfileBuild's target and the cleanup of the named Dockerfile; a parallel lifecycle command's header only on success; the 52-digit padding of the dev container id; a digest with a second colon; an empty request to the TS oracle. The final-stage label condition loses a clause that could never differ, and up's container metadata reader names the runtime lanes that run it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Mutation testing found that no test held node path.win32.basename's drive rule (only a letter and a colon are a drive) in winBasename. The 52-digit check of ComputeDevContainerID gets a mutate-exempt: < 53 pads nothing more. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Mutation testing found that no test held the stage-by-name lookup behind StageLabels, which keeps a user's metadata label of the targeted stage. Co-Authored-By: Claude Opus 5.5 <[email protected]>
TS ensureDockerfileHasFinalStageName reads only the physical last FROM line, so `FROM \` followed by `alpine AS base` becomes `FROM \ AS dev`, a Dockerfile no builder accepts. Ask BuildKit for that stage's name instead; every other FROM line keeps TS's handling. The ts-oracle golden records TS's answer as a divergence case, and docs/DIVERGENCES.md explains it. A `# escape=` directive now follows TS too, also recorded in the golden. Co-Authored-By: Claude Opus 5.5 <[email protected]>
mutate without Docker over this branch's diff left edits no test
refused. A win32 basename keeps `{:foo` whole, as node's
path.win32.basename does ('{' lowercases past 'z'); oracletest names
the JSON error of a bad recorded or Go result; a case's subtest name
takes the first of its name and fn#index. Dockerfile drops the escape
token nothing reads since base images resolve as TS does, finalStageName
leans on instructions.Parse returning no stages with its error, and the
two index edits that cannot change a match are exempt. stageLabel loses
a check that could not fail: on every FROM line BuildKit accepts, its
continuations joined, the FROM-line expression reads BuildKit's label.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
testdata:ts-oracle:check ran in no CI job, so a reference upgrade or a hand-edited golden went unseen. TestTSOracleGoldensMatchTheReference bundles the oracle and compares every golden with what the reference's functions return; task parity:contract, which CI runs after task reference, now runs it, and testdata:ts-oracle:check is that test. tsOracle keeps only the bundle and the node process, the lane test holding it; refreshTSOracleFiles reads, compares and writes, tested with a fake oracle. build.mjs resolves a symlinked reference/, since esbuild reports real paths. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Building a compose configuration now goes through `docker compose build`, as in TS. Compose builds through bake, and bake on buildx 0.38 and later refuses the Features Dockerfile, written under the cache folder outside the build context, unless --allow=fs.read is passed, which compose cannot do. TestE2E_ComposeBuildLabel failed that way on the runners. The compose processes this CLI starts get BUILDX_BAKE_ENTITLEMENTS_FS=0 unless the variable is already set. The reference fails there; docs/DIVERGENCES.md records it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
aledbf
force-pushed
the
test/differential-runtime
branch
from
October 26, 2026 00:09
619986b to
81a93af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The parity matrix, unit tests and e2e suites stayed green while the CLI still differed from the reference in ways nobody had written a case for. This PR adds two kinds of tests that take expected values from the reference itself rather than from hand-written cases, and fixes everything they found.
tools/tsoracle,internal/tsoracle). The oracle runs the reference's own functions fromreference/src, bundled with the reference's esbuild, over JSON cases.task testdata:ts-oraclerecords the results ininternal/*/testdata/ts-oracle, and the Go ports are compared against them. A case markeddivergencehas to keep differing, so fixing it means deleting the mark. About 480 cases cover the Dockerfile helpers,mergeConfigurationand the metadata label, the Feature build helpers, variable substitution anddevcontainerIdForLabels,getRef, the compose helpers, and more.TestDifferentialUp,TestDifferentialBuild, next toTestDifferentialReadConfiguration). Each one runsuporbuildon every upstream fixture with both CLIs and compares what a user sees: the JSON result, the container and image configuration (withdevcontainer.metadataparsed), mounts, andidinside the container. The known-differences list can only shrink. Ten entries remain, all the accepted legacy GitHub Release Feature limitation (5 fixtures × up/build).Bugs fixed
Each fix comes with a golden case, a unit test or a parity case that fails before the fix.
${devcontainerId}hashed labels with&<>and U+2028/U+2029 escaped and keys sorted by bytes, so some label sets produced a different id.AS Final), CRLF Dockerfiles gotASafter the\r, and aDevtarget was not found. Nested defaults like${IMG:-${REG}/app}, the unresolved-variable error,${...Basename}of/, and win32 trailing separators all differed too.&&and<>.readComposeVersionPrefixkept\r.parseMountcutsource=/a=bin the wrong place.mergeConfigurationtookrunArgsfrom metadata entries, where TS takes them from the config only.JSON.stringifywrites it. Containers are read back the waygetImageMetadataFromContainerreads them.buildgoes throughdocker compose buildwith the label override, so the image carries compose's labels and the project name followsgetProjectName.--secrets-filedoes not apply there (documented).containerEnvasENVlines, as build already did.FROM img-$TARGETARCHnames the right image.EnsureFinalStageNamenow returns TS's errors for a Dockerfile with no FROM, or with an unparseable final FROM.remoteUser. Before, it probed as the metadata user and gave the command that user's HOME and PATH. New runtime parity case:exec.config-remote-user-probe-success.--container-idcannot be inspected. New parity case:read-configuration.container-id-missing-error.Deliberate divergences (docs/DIVERGENCES.md, each recorded as a golden divergence case)
A final
FROMline continued with a backslash keeps its stage name. TS reads only the physical line and writesFROM \ AS dev, a Dockerfile no builder accepts. Here BuildKit reads the stage's name, and every other FROM line is handled exactly as TS handles it.Dockerfile ARG/ENV values are read with BuildKit's parser. Upstream fixes #1213 (colons in
${localEnv:VAR:default}) and #881 (:roin compose volumes) were already ported, and both are newer than the pinned reference. Object keys inside metadata label values are sorted.Compose builds may read the Features Dockerfile outside the build context. Now that a compose
buildgoes throughdocker compose build, bake on buildx ≥ 0.38 refused the Features Dockerfile, written in the cache folder, unless--allow=fs.readwas passed, which compose cannot do.TestE2E_ComposeBuildLabelfailed this way on the runners. The reference fails there too. This CLI starts its compose processes withBUILDX_BAKE_ENTITLEMENTS_FS=0unless that variable is already set.A
# escape=directive now follows TS (recorded in the golden).CI gate for the goldens
testdata:ts-oracle:checkwas not run by any CI job, so a stale golden would never be caught.TestTSOracleGoldensMatchTheReferencecompares every golden with what the reference returns now, andtask parity:contract, which CI runs aftertask reference, now runs it too. It caught a hand-edited golden in this branch.Validation
go vet ./...on every commitgo test -race ./...(Docker unreachable)task linttask parity:contract(incl. the new oracle gate)TestDifferentialReadConfigurationTestDifferentialUp/TestDifferentialBuild(Docker, local)go tool mutateover this diff, Docker unreachable,-j 1go tool mutate -j 4reported false survivors inextendImageWithFeatures, a different set on each run. Every one of them was refused when run serially, or when the mutation was applied by hand. Mutants that write into the package directory interfere with each other's runs; this is a go-tools follow-up, and it changes nothing here.🤖 Generated with Claude Code