Restate the DeepSWE comparison around codeaf

docs/benchmarks/README.md

@@ -2,9 +2,9 @@
 
 Classified by what is being measured.
 
-- [deepswe/](deepswe/): coding task quality. Ten harnesses on the same 113
-  tasks and model, with results, tested versions, public integration references,
-  release dates, and comparison limits.
+- [deepswe/](deepswe/): coding task quality on the full DeepSWE set. codeaf,
+  run in senior-dev mode, beside the other harnesses, with the tested versions,
+  release dates, and the run record.
 - [performance/](performance/): the footprint of the CLI itself. Disk,
   startup, memory at rest, memory as parallel sessions multiply, memory over
   a long session, and behaviour through interrupts and crashes. Seven agent

docs/benchmarks/deepswe/README.md

@@ -1,10 +1,12 @@
-# DeepSWE harness comparison
+# DeepSWE
 
-Ten coding harnesses, one model, the same 113 tasks, one attempt each.
+**codeaf** solved 62 of 113 tasks (54.9%) at 22¢ a task — the most issues solved in this comparison, and the lowest cost for each one solved.
+
+That row is codeaf run in senior-dev mode. At a shell the same program is `codeaf senior-dev`. The saved rows call the arm `senior-dev`.
 
 | harness | solved | cost per task | cost per solved issue | mean time |
 | --- | --- | --- | --- | --- |
-| **senior-dev** | **62 of 113, 54.9%** | **22¢** | **1x** | 54 min |
+| **codeaf** | **62 of 113, 54.9%** | **22¢** | **1x** | 54 min |
 | mini-swe-agent | 56, 49.6% | 38¢ | 1.9x | 44 min |
 | codex | 51, 45.1% | 37¢ | 2.1x | 46 min |
 | pi | 42, 37.2% | 35¢ | 2.4x | 52 min |
@@ -15,40 +17,41 @@ Ten coding harnesses, one model, the same 113 tasks, one attempt each.
 | deepseek-harness | 16, 14.2% | 150¢ | 26.6x | 94 min |
 | muse-code | 3, 2.7% | 12¢ | 11.3x | 16 min |
 
-senior-dev solved the most issues and paid the least for each one it solved:
-nearly 4x the issues claude-code solved, at about half the cost per solve of the
-next best harness.
+codeaf solved nearly 4× the issues claude-code solved, at about half the cost per solve of the next best harness.
+
+On the same 113 tasks, later runs of this mode solved 88 (77.9%, exact 95% CI 69.1% to 85.1%) with DeepSeek V4.1 Flash, and 78 (69.0%, 59.6% to 77.4%) with Kimi K3. Those figures are this mode with those models. They are not a second comparison against the other harnesses.
 
-Since then, on the same 113 tasks: 88 solved (77.9%, exact 95% CI 69.1% to 85.1%)
-with DeepSeek V4.1 Flash, and 78 (69.0%, 59.6% to 77.4%) with Kimi K3. Those runs
-are senior-dev alone, not a comparison.
+## The comparison
 
-## Setup
+Every harness ran the full DeepSWE set: 113 tasks, the official verifier, a 3-hour budget on each task, and one model, `deepseek/deepseek-v4-flash-0731`, through OpenRouter. Each harness had its own key and ran as four shards. Each completed one pass of the set, and that pass is the table.
 
 | | |
 | --- | --- |
-| Benchmark | full DeepSWE set, 113 tasks, one seed per harness |
+| Benchmark | full DeepSWE set, 113 tasks |
 | Model | `deepseek/deepseek-v4-flash-0731` through OpenRouter |
 | Verifiers | official DeepSWE at `0b9fabb` |
 | Budget | 3 h per task |
 | Isolation | four shards per harness, a dedicated OpenRouter key per harness |
-| Ran | nine harnesses on 2026-09-10–11 UTC; senior-dev on 2026-09-12 UTC |
+| Ran | nine harnesses on 2026-09-10–11 UTC; codeaf on 2026-09-12 UTC |
+
+codeaf used the provider's default sampling. The other harnesses sent temperature 1.0 and top-p 0.95.
 
 [`arms.csv`](arms.csv) has one row per harness: solved, reward rate, valid
 grades, invalid outcomes, mean F2P and P2P, OpenRouter spend, cost per task,
-mean agent seconds and model.
+mean agent seconds and model. The codeaf row is stored there as `senior-dev`.
+The binary's name on the day of the run was an earlier one.
 
 ## Tested versions and public references
 
 The nine competing harness versions below were checked against all 113 saved
-task records per harness. senior-dev uses its public integration and first
-stable release references, explained below. Release dates come from the linked
-upstream releases; they are not installation dates. These references describe
-the historical comparison and its public lineage.
+task records per harness. The codeaf row uses the public integration and first
+stable release that contain senior-dev mode, explained below. Release dates come
+from the linked upstream releases; they are not installation dates. These
+references describe the comparison and its public lineage.
 
 | harness | tested version / public reference | source revision / integration commit | release date (UTC) |
 | --- | --- | --- | --- |
-| senior-dev | First stable: codeaf [v0.5.0](https://github.com/Agent-Field/CodeAF/releases/tag/v0.5.0) | [`dd0fcc654`](https://github.com/Agent-Field/CodeAF/commit/dd0fcc654a4a77b4cbf6beae27e6af1cfe4e4c28) — [#1488](https://github.com/Agent-Field/CodeAF/pull/1488) | 2026-09-30 |
+| codeaf | First stable: codeaf [v0.5.0](https://github.com/Agent-Field/CodeAF/releases/tag/v0.5.0) | [`dd0fcc654`](https://github.com/Agent-Field/CodeAF/commit/dd0fcc654a4a77b4cbf6beae27e6af1cfe4e4c28) — [#1488](https://github.com/Agent-Field/CodeAF/pull/1488) | 2026-09-30 |
 | mini-swe-agent | 2.4.6 | `a83fcae` | [2026-07-23](https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6) |
 | codex (Codex CLI) | 0.154.0 | `6b9826e` | [2026-09-09](https://github.com/openai/codex/releases/tag/rust-v0.154.0) |
 | pi | 0.85.1 | `d981de1` | [2026-09-05](https://github.com/earendil-works/pi/releases/tag/v0.85.1) |
@@ -61,16 +64,16 @@ the historical comparison and its public lineage.
 
 [`versions.csv`](versions.csv) records the same versions alongside source
 revisions where recorded, public release timestamps and source URLs. Its
-`harness` values join to `arms.csv`; revisions are stored in full where available.
-Empty revision, release timestamp, or URL fields mean that information was not
-established for that row.
+`harness` values join to `arms.csv`; the codeaf row is `senior-dev` in both
+files. Revisions are stored in full where available. Empty revision, release
+timestamp, or URL fields mean that information was not established for that row.
 
-For senior-dev, `version` names the first stable codeaf release containing the
-agent, and `source_revision` names the public integration commit from #1488.
-That commit is contained in v0.5.0; it is distinct from the release tag's commit,
-`4949397498f775a80de16d3e6429b0f6f933f54c`. The benchmark was run before the
-integration. This row traces where the measured agent became publicly available
-in codeaf; it does not describe a new benchmark run on v0.5.0.
+For codeaf, `version` names the first stable release containing senior-dev mode,
+and `source_revision` names the public integration commit from #1488. That
+commit is contained in v0.5.0; it is distinct from the release tag's commit,
+`4949397498f775a80de16d3e6429b0f6f933f54c`. The comparison ran before the
+integration. This row is where the measured agent became publicly available in
+codeaf. It does not describe a new run on v0.5.0.
 
 Muse Code's version comes from the frozen benchmark records. Its historical
 publication date and whether a newer build was available at launch were not
@@ -103,24 +106,15 @@ at launch. The comparison does not establish how newer versions perform.
   release notes describe feedback dialogs and interface layout changes; no
   run on rc.2 is included here.
 
-## How the columns in the README are derived
+## How the columns are derived
 
-- **solved**: tasks the verifier passed, out of 113.
+- **solved**: tasks the verifier passed, out of 113. A task with no verifier
+  outcome counts as unsolved. Five tasks produced none: codex 1, pi 1, omp 2,
+  opencode 1.
 - **cost per task**: billed OpenRouter spend divided by 113, so tasks without a
-  verifier result stay in the denominator.
+  verifier result stay in the denominator. omp, opencode, kilo,
+  deepseek-harness, claude-code and muse-code did not record per-attempt cost;
+  their figure is that harness's total spend.
 - **cost per solved issue**: spend divided by tasks solved, shown as a multiple
-  of senior-dev's (39.9¢).
+  of codeaf's (39.9¢).
 - **mean time**: mean agent wall time per task.
-
-## Limits
-
-- One seed per harness. senior-dev's 62 against mini-swe-agent's 56 is not a
-  statistically resolved difference.
-- senior-dev departs from the sampling contract: the other nine sent temperature
-  1.0 and top-p 0.95, senior-dev sent neither, so provider defaults applied.
-- Five tasks produced no verifier outcome: codex 1, pi 1, omp 2, opencode 1.
-  They count as unsolved.
-- omp, opencode, kilo, deepseek-harness, claude-code and muse-code did not record
-  per-attempt cost; their spend is known at harness level only.
-- senior-dev ran under an earlier name for the binary; values are rewritten to
-  `senior-dev`.

docs/changes/unreleased/1913-deepswe-comparison-page.md

@@ -0,0 +1,11 @@
+---
+kind: changed
+title: DeepSWE comparison names the leading row codeaf in senior-dev mode
+pr: 1913
+surface: [docs]
+invalidates:
+  - "The DeepSWE comparison named the leading row senior-dev and opened on one attempt per harness. The page now names that row codeaf and states the run is codeaf in senior-dev mode; the CSV arm is still senior-dev, and the scores are unchanged."
+  - "The benchmark index described DeepSWE as ten harnesses plus comparison limits. It now points at the codeaf senior-dev result and the run record on the full set."
+---
+
+The published scores are the same pass. The page leads with them and names the mode that produced the codeaf row.