Repository navigation
Add workflow-eval run local to run local-inference Runs without a browser tab - #563
Draft
joaomarcoscrs wants to merge 17 commits into
Draft
joaomarcoscrs wants to merge 17 commits into
joaomarcoscrs wants to merge 17 commits into
Conversation
…rowser tab
Runs whose inference server is local (executionMode "browser") can now be
driven by the CLI in the background. The runner reads input-cases (with the
server-resolved workflowInputs), calls {url}/workflows/run with the same body
and headers as the web app's worker, and submits each output through the
existing capture route so the server keeps scoring.
- Sliding window of --concurrency inferences (default 4, max 16); capture
POSTs run outside the inference slots (timeout 300 s), with backpressure
and at most one page fetched ahead; stale pages are re-read for fresh
media URLs before dispatch.
- Cases with workflowInputFailures are not sent to the inference server; they
are captured as {output: null, runtime: {error}} like the engine does.
- Retry up to 3 attempts on network/408/429/5xx before submitting; stop after
10 consecutive connection failures. Preflight POST /workflows/blocks/describe.
- Capture responses: 409 different/processing/non-retryable/closed, 412
attempt changed, 413 replaced by an explicit failure capture, 404/400.
- Local journal (journal.jsonl + captures/<sha256>.json, fsync per line) so a
rerun after kill -9 resubmits the same bytes instead of re-running inference.
- SIGINT/SIGTERM: stop dispatching and drain; a second signal exits now; 130.
- `run start --wait --inference-url` runs the local loop in-process and waits.
- CLI-COMMANDS.md quickstart.
…n instead of failing on server Runs - Connection-level failures (refused/reset, DNS, connect timeout, connection dropped mid-response) are infrastructure: after the 3 attempts nothing is captured or journaled, the Case stays pending and still counts toward the 10-consecutive-failures stop. A run that leaves Cases pending exits 1 with a resume hint. Read timeouts and HTTP 408/429/5xx keep the retryable failure capture. - 409 "No retained capture" on a resume and a page at another attemptCount restart the Execution like 412; tests prove no duplicate capture. Capture response table in the runner docstring. - `run start --wait --inference-url` on a server Run warns that the URL has no effect and keeps the normal wait. - `--case-timeout` help: per connection attempt and per read, not a total. - With --json, warnings go in a "warnings" array of the final JSON.
…nate a torn journal tail before appending - The second SIGINT/SIGTERM raises from the signal handler, so a blocking call on the main thread (get_run, configuration, preflight, page fetch) is interrupted and the CLI exits 130; an aborted run never reports success. - Opening a journal whose last line was torn writes a newline first, so the next record is not glued to it and lost on the next restart.
…d stop retrying read timeouts - run start --wait --inference-url: every local runner error gets the run local resume hint and the runId, so a rerun does not start a second Run. - A Run with cancellationRequestedAt that is not terminal yet counts as cancelled (exit 1). - A read timeout is one timedOut capture, like the browser; only connection/408/429/5xx are retried. - --json errors keep the collected warnings; a transient error on run local's final Run read keeps the last known state. - An output that cannot be serialized becomes a non-retryable failure capture instead of stopping the run. - A prefetch abandoned by an internal reload ends the pass instead of exiting 130; stale prefetches stop with their pass. - Docstrings: resubmission keeps the JSON value, first stop abandons retries in backoff.
…ting 0 A transient error on the last get_run now backs off like every other Roboflow API call (up to 60 s, 10 min budget) and exits 1 when the budget runs out; exit 0 still requires the Run's state to be read.
- After run start --wait --inference-url created the Run, runner hints drop "run the command again" and end with the run local resume command; a cancelled Run gets no resume hint. - Unexpected errors from the local loop (e.g. an unwritable state dir) also carry runId and the resume command. - --json keeps collected warnings on the generic error path; CLI-COMMANDS.md documents warnings and error.warnings.
…un local does not wrap other errors
Contributor
Author
|
[Jarbas Local João] — APPROVED at Gist: https://gist.github.com/joaomarcoscrs/8df231d66fd43e06357848736d8f4faa |
Contributor
Author
|
[Jarbas Local João] — APPROVED at Gist: https://gist.github.com/joaomarcoscrs/6084637cb2b136d39a5e14864cf23690 |
… refresh URLs before retries - A connection dropped after it was made (reset, aborted, closed mid-response) is retried like HTTP 5xx and then captured as retryable; only a request that could not connect leaves the Case pending, after waiting 1, 2, 4, 8 and 16 s. - The limit of 10 counts consecutive Cases that could not connect, not HTTP attempts; any attempt that reaches the server resets it. - Before a retry, a Case whose input-cases page is older than 8 minutes re-reads it, so the retry gets fresh signed media URLs. - The run start resume hint repeats --workspace, --concurrency and --case-timeout when they were passed.
The re-read now stops retrying once dispatching stops (halt, reload or closed Run), and a halt that lands during the re-read no longer starts another inference.
Contributor
Author
|
[Jarbas Local João] — APPROVED at Gist: https://gist.github.com/joaomarcoscrs/c40601f410dc254fd1f012c7d703f6a1 |
…ody as dropped connections - A Case missing from the page re-read before a retry now flags a reload, so the Execution restarts and finds it instead of leaving it pending. - The inference response is streamed: SSLError/ProxyError while sending stay "could not connect", while any connection error reading the body counts as a dropped connection (3 attempts, then a retryable capture). - Document the worst case per Case: about 31 s of connect backoff plus 3 attempts x --case-timeout.
Each connection attempt can itself wait --case-timeout, and the timeout limits each read rather than the whole response, so the bound is about 34 s of backoff plus 9 x --case-timeout for a server that stops responding.
A connected attempt can wait --case-timeout to connect and again on a read, so the bound is about 34 s of backoff plus 11 x --case-timeout.
Contributor
Author
|
[Jarbas Local João] — APPROVED at Gist: https://gist.github.com/joaomarcoscrs/f94520f4f567a3382e2d9ae5423718be |
…hile waiting for a lease - A capture larger than 10 MiB as requests sends it (json.dumps with ensure_ascii, not the stored bytes) is replaced by the failure capture of a 413 without being sent; the server limit is setupExpress.ts:77. - While waiting for another runner's lease, the Case page is re-read every 5 s: a closed Run or a new attempt ends the wait (the run exits 0 when the Run completed), a completed or failed Case is resent at once, and a changed leaseExpiresAt moves the deadline. _lease() is gone.
Contributor
Author
|
[Jarbas Local João] — APPROVED at Gist: https://gist.github.com/joaomarcoscrs/844c221cbaaf7ed0b74e3ed596e4ada7 |
…ait restarts the Execution
Contributor
Author
|
[Jarbas Local João] — APPROVED at Gist: https://gist.github.com/joaomarcoscrs/c03fbb27a9feb1976954a0722985fa9e |
|
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New
roboflow workflow-eval run local <run-id> -e <eval-id>(Run id positional, likerun get/cancel/retry <run-id>): runs a local Run (executionMode: browser, inference server such ashttp://localhost:9001) to the end in the background, without a browser tab. The server keeps scoring.input-cases(with the server-resolvedworkflowInputs), calls{url}/workflows/runwith the same body and headers as the web app's worker, and submits each output through the existingcaptureroute.--concurrencyinferences (default 4, the browser's limit; 1-16). Capture POSTs run outside the inference slots (timeout 300 s, scoring runs inside the request), with backpressure and at most one page fetched ahead. Stale pages are re-read for fresh media URLs before dispatch.workflowInputFailuresare not sent to the inference server; they are captured as{output: null, runtime: {error}}, like the engine does.retryable: truefailure capture. A request that could not connect (refused, DNS, connect timeout, TLS/proxy while sending) is retried after 1, 2, 4, 8 and 16 s; if it still cannot connect, no capture is sent and the Case stays pending. 10 consecutive Cases that could not connect stop the CLI (exit 1, journal kept); any attempt that reaches the server resets the count. A run that leaves Cases pending exits 1 with a resume hint. Before each retry, a Case whose page is older than 8 min re-reads it for fresh media URLs.journal.jsonl+captures/<sha256>.json, fsync per line) so a rerun afterkill -9resubmits the same bytes instead of re-running inference.POST /workflows/blocks/describe. SIGINT/SIGTERM stop dispatching and drain; a second signal exits now; exit 130 with the resume command.run start --wait --inference-url URLruns the local loop in-process and then waits. Its resume hint repeats--workspace,--concurrencyand--case-timeoutwhen they were passed. If the Run turns out to execute on Roboflow, it warns that--inference-urlhas no effect and keeps the normal wait.--json, warnings go in awarningsarray of the final JSON; stdout stays one JSON document.CLI-COMMANDS.mdquickstart.Capture responses
done_by_othercompleted/failed, follow a changedleaseExpiresAt, stop (exit 0 if the Run completed) if the Run closed or the attempt changed; then resend the same bytes. The same wait applies before aresumeof a capture the server retainedresume)input-casespage with a differentattemptCountWorkflow output exceeds the 10 MB capture limit(retryable: false). The CLI also checks the size before sending (the JSON asrequestssends it, over 10 MiB, the server'sbodyLimitinsetupExpress.ts:77) and sends the failure capture directlyWhy
F1 of "evals on device": local runs (about 3 h on a local inference server for slow workflows) must run in the background from the CLI, at least as fast as the browser run, and survive interruptions without duplicate captures.
How tested
pytestat 61e029d: 1405 passed, 1 skipped.ruff format --check,ruff check,mypy roboflow.input-caseschange below deployed to staging. End-to-end validation (long run,kill -9resume, metric and timing parity with the browser, EU) happens before this leaves draft.Related (F1)
--waitwithout a time limit.resolveWorkflowCaseInputs.lastActivityAton browser-mode Runs.input-caseswithworkflowInputs/workflowInputFailures: PR not opened yet, waits for the engine 0.5.1 release.Known limitations
input-caseschange in staging/production; against an older server the CLI exits 1 with "server does not support local runner yet".--case-timeoutapplies per connection attempt and per read (requests semantics), not as a total deadline per Case. A total deadline is a follow-up.requestsneeds a minimum version of 2.27: below that, anSSLErrorraised while reading the response body escapes unhandled. This was already true before this PR and is not blocking; follow-up.