Repository navigation
Let workflow-eval --wait run without a time limit and show progress - #562
Draft
joaomarcoscrs wants to merge 1 commit into
Draft
joaomarcoscrs wants to merge 1 commit into
joaomarcoscrs wants to merge 1 commit into
Conversation
- wait_for_run(timeout=None or 0) waits without limit; positive timeouts still raise TimeoutError and the SDK default stays 3600 s. - run get/start --wait: --timeout 0 means no limit; without --timeout, Runs on a local inference server (executionMode browser) wait without limit and server Runs keep 3600 s. - Print a progress line on stderr at most every 10 s while waiting, suppressed with --json and --quiet. - submit_capture: runtime is required in the body; accept an optional HTTP timeout because scoring runs inside the request.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
Author
|
[Jarbas Local João] — APPROVED at Gist: https://gist.github.com/joaomarcoscrs/1c89efa727628a195da6e6eb47768ed2 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
wait_for_run(timeout=None or 0)waits without limit; positive timeouts still raiseTimeoutError, SDK default stays 3600 s.workflow-eval run get/start --wait:--timeout 0means no limit; without--timeout, Runs on a local inference server (executionMode: browser) wait without limit and server Runs keep 3600 s.--json/--quiet).submit_capture:runtimerequired in body; optional HTTP timeout since scoring runs inside the request.CLI-COMMANDS.mdupdated.Why
F1 of "evals on device": local runs (~3 h on a local inference server) must be driven by the CLI in the background, without the browser and without hitting the 1 h wait cap.
How tested
pytest(61 passed, 1 skipped),ruff format --check,ruff check,mypy.Related (F1)
resolveWorkflowCaseInputshttps://github.com/roboflow/workflow-evals/pull/63lastActivityAton browser-mode Runs https://github.com/roboflow/roboflow/pull/18280