Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
a6ba783
Add `workflow-eval run local` to run local-inference Runs without a b…
joaomarcoscrs Oct 22, 2026
7b0fe56
Leave Cases pending when the inference server is unreachable, and war…
joaomarcoscrs Oct 22, 2026
98e8adb
fix(pre_commit): 🎨 auto format pre-commit hooks
pre-commit-ci[bot] Oct 22, 2026
bc15d45
Abort at once on a second signal during blocking API calls, and termi…
joaomarcoscrs Oct 22, 2026
3d8af09
Keep the Run id in run start errors, report pending cancellations, an…
joaomarcoscrs Oct 22, 2026
c80b034
Retry run local's final Run read within the API budget instead of exi…
joaomarcoscrs Oct 22, 2026
dad64eb
Point every run start failure at run local, never at rerunning run start
joaomarcoscrs Oct 22, 2026
6a18ed8
Scope the run start resume promise to the local loop, and test that r…
joaomarcoscrs Oct 22, 2026
509df11
Take the Run id positionally in run local, like the rest of the run s…
joaomarcoscrs Oct 22, 2026
819784e
Capture dropped inference connections, back off unreachable ones, and…
joaomarcoscrs Oct 22, 2026
ed813aa
Abandon a retry's page re-read when the runner halts or the pass reloads
joaomarcoscrs Oct 22, 2026
fffadc1
Keep the runner tests collectable on urllib3 1.26, which has no NameR…
joaomarcoscrs Oct 22, 2026
085c678
Reload when a Case leaves its re-read page, and count TLS drops mid-b…
joaomarcoscrs Oct 22, 2026
a5e7a06
State the per-Case worst case with every attempt the runner can make
joaomarcoscrs Oct 22, 2026
8f6c0a6
Count both timeouts of a connected attempt in the per-Case worst case
joaomarcoscrs Oct 22, 2026
b082373
Skip captures over the 10 MiB body limit, and re-read the Case page w…
joaomarcoscrs Oct 22, 2026
61e029d
Test that a Case leaving its page, or a new attempt, during a lease w…
joaomarcoscrs Oct 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CLI-COMMANDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,25 @@ roboflow workflow-eval binding suggest --spec <spec-id> --dataset <dataset-id> -
roboflow workflow-eval run start --eval <eval-id> --body @run.json --wait
# --wait gives up after 3600 s; Runs on a local inference server wait without limit. --timeout 0 = no limit.
roboflow workflow-eval run get <run-id> --eval <eval-id> --wait --timeout 0

# Runs on a local inference server (executionMode "browser") run without a browser tab.
# Start and run one in this process, then wait for the result:
roboflow workflow-eval run start --eval <eval-id> --body @run.json --wait --inference-url http://localhost:9001
# Or run an existing local Run in the background (default --concurrency 4, max 16):
nohup roboflow workflow-eval run local <run-id> -e <eval-id> --concurrency 4 > run-local.log 2>&1 &
# Interrupted (Ctrl-C, exit 130)? Run the same command again: captured outputs are resubmitted from the
# journal in ~/.cache/roboflow/workflow-evals/<workspace>/<run-id>/ (--state-dir), not re-run.
# Inference server unreachable (cannot connect, retried after 1, 2, 4, 8, 16 s)? Those Cases are not captured
# (they stay pending; PENDING in the summary) and the command exits 1, at once after 10 such Cases in a row;
# start the server and run it again. A read timeout (--case-timeout: per connection attempt and per read, not
# per Case), or a 408/429/5xx or dropped connection that persists for 3 attempts, is captured as a retryable
# failure. Worst case per Case for a server that stops responding: about 34 s of backoff plus 11 x --case-timeout
# (--case-timeout limits each connect and each read, and a Case makes up to 6 connection attempts and 3 connected
# ones); a response that keeps sending data is not cut off (no bound with --case-timeout 0). With --json,
# warnings are in "warnings" on success, or in "error.warnings" on failure.
# If `run start --wait --inference-url` fails while running the Cases locally, the error has the Run id
# (error.details.runId) and, unless the Run was cancelled, a `run local` command (with the --workspace,
# --concurrency and --case-timeout you passed) to resume it; running `run start` again would start another Run.
roboflow workflow-eval execution overview <execution-id> --eval <eval-id> --run <run-id> --json
roboflow workflow-eval execution results <execution-id> --eval <eval-id> --run <run-id> --failed-check
roboflow workflow-eval compare --eval <eval-id> -x <execution-a> -x <execution-b> --json
Expand Down
Loading