[yuyixg]

A run executed its cases one after another, so a suite cost the sum of every case's model wait: about 25 minutes for the 39-case suite whose cases each take 30 to 50 s, nearly all of it spent waiting.

Cases are independent by construction. Each one opens its own conversation and its own DI scope, and the mock seam, the model override and the tool recorder all find their case by conversation id in a ConcurrentDictionary, never by an ambient "current case" -- so two cases in flight cannot see each other. The executor now runs up to MaxParallelCases work items at a time, taken from the run's own value, else the suite's (default 4, 1 to 16, rejected rather than clamped at the API).

What must not overlap is this layer's own bookkeeping: the run document's counts are a read-modify-write of one document, and two of those interleaved lose whichever increment landed first. Each case's tally is therefore taken under a semaphore while only the case executions themselves overlap, which costs one round trip against a case's tens of seconds.

Cancelling now distinguishes "stop starting new work" from "the run ended early because of it". A cancel that lands while the last case is executing holds nothing back, so that run still ends in its real status rather than being retroactively Cancelled. Cases already in flight cannot be aborted -- BotSharp's SendMessage takes no CancellationToken -- so they finish and record instead of losing work already paid for.

The run records the concurrency it ran at, because a latency percentile from a four-at-a-time run is not the same measurement as one from a serial run, and the executor logs one line per finished case: interleaved cases make progress unreadable from a case's own log lines.

[qodo-code-review]

Code Review by Qodo

Sorry, something went wrong

We weren't able to complete the code review on our side. Please try again manually by commenting /agentic_review on this PR.

[qodo-code-review]

PR Summary by Qodo

Run agent test cases concurrently with configurable limits

✨ Enhancement 🧪 Tests 🕐 40+ Minutes

AI Description

• Run independent suite cases concurrently to reduce time spent waiting on models.
• Support validated suite limits and per-run overrides, and record the effective limit on each run.
• Preserve accurate counts and cancellation outcomes while adding progress logs and concurrency
 tests.

Diagram

graph TD
  API["Test API"] --> Settings["Concurrency settings"] --> Executor["Run executor"] --> Cases["Scoped case runners"] --> Bookkeeping["Serialized bookkeeping"] --> Results[("Case results")]
  Bookkeeping --> Run[("Run document")]
  Executor --> Run

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Atomic database counter updates

• ➕ Could protect counts across multiple executor processes without an in-process semaphore.
• ➖ Requires repository changes and separate coordination for cancellation and other run-document fields.

Recommendation: Keep bounded concurrency in the executor and serialize bookkeeping for the current single-consumer run queue. Consider atomic repository updates if runs may later execute across multiple processes.

Files changed (7) +544 / -59

Enhancement (5) +236 / -48

AgentTestController.csValidate and accept suite and run concurrency limits +32/-1

Validate and accept suite and run concurrency limits

• Rejects requested limits outside 1–16 for suite creation, suite updates, and run triggers. Preserves an existing suite limit when an update omits it and stores an omitted run override as a fallback sentinel.

src/Plugins/BotSharp.Plugin.AgentTesting/Controllers/AgentTestController.cs

AgentTestDtos.csExpose optional concurrency settings in requests +17/-0

Expose optional concurrency settings in requests

• Adds nullable MaxParallelCases fields to suite upsert and run trigger requests, allowing omitted values to retain suite settings or use suite fallback.

src/Plugins/BotSharp.Plugin.AgentTesting/Models/AgentTestDtos.cs

AgentTestRun.csRecord effective concurrency on runs +13/-0

Record effective concurrency on runs

• Adds MaxParallelCases to the run document so performance results can be interpreted alongside the concurrency used.

src/Plugins/BotSharp.Plugin.AgentTesting/Models/AgentTestRun.cs

AgentTestSuite.csDefine suite concurrency defaults and resolution +48/-0

Define suite concurrency defaults and resolution

• Adds a suite limit defaulting to four and centralizes fallback and ceiling rules. Resolution favors a positive run override, then the suite setting, and clamps the effective limit to 1–16.

src/Plugins/BotSharp.Plugin.AgentTesting/Models/AgentTestSuite.cs

AgentTestRunExecutor.csExecute bounded parallel cases with serialized bookkeeping +126/-47

Execute bounded parallel cases with serialized bookkeeping

• Replaces the serial loop with bounded parallel work while protecting result writes and run-count updates with a semaphore. Tracks skipped work separately from cancellation requests, preserves in-flight results, records effective concurrency, and logs per-case progress.

src/Plugins/BotSharp.Plugin.AgentTesting/Services/AgentTestRunExecutor.cs

Tests (2) +308 / -11

AgentTestControllerTests.csCover concurrency defaults, overrides, and API rejection +98/-2

Cover concurrency defaults, overrides, and API rejection

• Tests suite defaults, partial-update preservation, run override persistence, and invalid-limit rejection. Adjusts existing creation assertions for the controller's ActionResult return type.

tests/BotSharp.Core.UnitTests/AgentTesting/AgentTestControllerTests.cs

AgentTestRunExecutorTests.csVerify parallel limits, counts, fallback, and cancellation +210/-9

Verify parallel limits, counts, fallback, and cancellation

• Pins order-dependent tests to serial execution and adds probes for overlapping cases and concurrency limits. Tests fallback resolution, clamping, exact tallies, and cancellation that skips unstarted work.

tests/BotSharp.Core.UnitTests/AgentTesting/AgentTestRunExecutorTests.cs

[yileicn]

reviewed