Skip to content

fix(tools): bound ExecuteBashTool stdout/stderr capture - #7521

Open
Hugo-DDT wants to merge 5 commits into
google:mainfrom
Hugo-DDT:fix/bash-tool-bounded-output
Open

Hugo-DDT wants to merge 5 commits into
google:mainfrom
Hugo-DDT:fix/bash-tool-bounded-output

Conversation

@Hugo-DDT

@Hugo-DDT Hugo-DDT commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

ExecuteBashTool.run_async() collected subprocess output with process.communicate(), which buffers all of stdout and stderr in the agent (parent) process with no size limit. RLIMIT_AS / RLIMIT_FSIZE apply to the child, not to the pipes, so a user-approved but verbose command could use up the agent process's memory before the timeout fires. The timeout path also returned everything it had captured, with no cap.

Fix

  • New BashToolPolicy.max_output_bytes_per_stream (default 1 MiB, None disables).
  • stdout and stderr are read concurrently in 64 KiB chunks into per-stream bounded buffers. Bytes over the limit are still read but thrown away, so the child never blocks on a full pipe, and memory stays at roughly limit + one chunk per stream.
  • Truncated output ends with an explicit marker, e.g. ...[stdout truncated: N of M bytes omitted; limit is L bytes].
  • Timeout and cancellation use an explicit waiter. Readers and asyncio.create_task(process.wait()) are tracked together, and asyncio.wait returns when all three finish or the timeout fires. On timeout the process group is killed, then _finish_tasks drains for a short grace period and cancels anything still pending. On cancellation those tasks are cancelled and gathered before CancelledError is re-raised. An unexpected execution error uses the same finish step. A setsid descendant that keeps a pipe open can no longer leave process.wait() pending after run_async returns.
  • After that cleanup, the subprocess transport is closed with process._transport.close() when it is present. killpg does not reach a descendant that called setsid() and kept the inherited pipe, so the writer would otherwise keep running. Closing the transport drops the read end and the descendant exits on EPIPE. AttributeError, OSError, and RuntimeError from a missing or already-closed transport are ignored.

Tests

tests/unittests/tools/test_bash_tool.py:

  • small output comes back unchanged
  • 2 MiB of stdout is truncated to the limit, with the marker and the correct omitted-byte count
  • interleaved large stdout and stderr are each bounded on their own
  • None turns the limit off
  • a command that never ends and writes nonstop times out, its output is bounded, and the process is reaped (returncode == -SIGKILL)
  • a setsid background child that keeps the pipe open does not leave drain/wait tasks pending, and that writer process exits after the transport is closed
  • cancelling run_async does not leave drain/wait tasks pending, and a setsid writer on the cancel path exits as well
  • cancelling run_async does not report an unretrieved task exception
  • output exactly at the limit is not marked truncated
  • max_output_bytes_per_stream < 0 is rejected
  • a transport close() that raises AttributeError or OSError still returns the timeout error

Existing mocked-process tests stub stdout/stderr instead of communicate(). tests/unittests/tools/test_bash_tool_windows.py still covers the non-POSIX error.

pytest tests/unittests/tools/test_bash_tool.py tests/unittests/tools/test_bash_tool_windows.py → 42 passed on Python 3.12. Formatted with pyink + isort.

Fixes #7507

Hugo-DDT added a commit to Hugo-DDT/adk-python that referenced this pull request Oct 15, 2026
Address review on google#7521 / google#7507: on CancelledError kill the process
group, cancel completion and drain readers, gather them, then re-raise;
reject max_output_bytes_per_stream < 0; add exact-at-limit and
cancellation tests.

@BichengWang BichengWang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 9cf04517672d6b3f2a868ba2d5fc3eb81bf40234, including the cancellation follow-up discussed in #7507.

[P2] Retrieve the original completion future after cancellation — bash_tool.py:385–390

completion.cancel() cancels the original gather(), but the cleanup only awaits a different gather over readers. Since wait_for() has already cancelled the shield, nothing retrieves completion's CancelledError. A normal, handled cancellation consequently reaches the event loop's exception handler as:

_GatheringFuture exception was never retrieved
asyncio.exceptions.CancelledError

This produces an unhandled-error log and invokes application exception-reporting hooks for expected cancellation. It reproduces with a single sleeping subprocess; no background descendant is needed. The existing cancellation test checks pending reader tasks, so it passes despite this error.

Please await/consume the original completion future after cancelling it, while preserving cancellation propagation. A regression can install a temporary loop exception handler, start a command that writes a readiness marker and sleeps, wait for the marker, cancel run_async, catch its expected CancelledError, yield/collect finished futures, and assert that the handler received no errors. I verified that awaiting asyncio.gather(completion, return_exceptions=True) after completion.cancel() clears this reproduction. The timeout path also needs to consume completion when _finish_readers() cancels readers: a detached descendant holding a pipe open reproduced the same warning there.

Validation on macOS / Python 3.13.16, offline:

  • PYTHONPATH=src python -m pytest -q -p no:cacheprovider tests/unittests/tools/test_bash_tool.py tests/unittests/tools/test_bash_tool_windows.py: 37 passed.
  • The real-subprocess cancellation regression passes against the PR base 62745ccc62513514bc0f30f5cfdaf1c593d156ca, fails against this head with the error above, and passes with the completion-await remedy applied only to an in-memory module copy.
  • Feeding 64 MiB in 64 KiB chunks into _BoundedOutput with a 1 MiB limit retained exactly 1,048,576 bytes, reported 66,060,288 omitted bytes, and peaked at 1,106,090 traced bytes during accumulation. This supports the bounded-capture approach independently of the returned string length.

I did not run the full multi-version tox suite.

@Hugo-DDT

Copy link
Copy Markdown
Contributor Author

Thanks @BichengWang — addressed in e8581a2: the cancelled completion future is now awaited so its CancelledError is retrieved while cancellation still propagates, and I added the regression test with a temporary loop exception handler as you described.

@Hugo-DDT
Hugo-DDT force-pushed the fix/bash-tool-bounded-output branch from e8581a2 to 78ed767 Compare October 16, 2026 08:56
Hugo-DDT added a commit to Hugo-DDT/adk-python that referenced this pull request Oct 16, 2026
Address review on google#7521 / google#7507: on CancelledError kill the process
group, cancel completion and drain readers, gather them, then re-raise;
reject max_output_bytes_per_stream < 0; add exact-at-limit and
cancellation tests.
@Hugo-DDT
Hugo-DDT force-pushed the fix/bash-tool-bounded-output branch from 78ed767 to c9aff13 Compare October 20, 2026 11:18
Hugo-DDT added a commit to Hugo-DDT/adk-python that referenced this pull request Oct 20, 2026
Address review on google#7521 / google#7507: on CancelledError kill the process
group, cancel completion and drain readers, gather them, then re-raise;
reject max_output_bytes_per_stream < 0; add exact-at-limit and
cancellation tests.
Hugo-DDT added a commit to Hugo-DDT/adk-python that referenced this pull request Oct 22, 2026
Address review on google#7521 / google#7507: on CancelledError kill the process
group, cancel completion and drain readers, gather them, then re-raise;
reject max_output_bytes_per_stream < 0; add exact-at-limit and
cancellation tests.
@Hugo-DDT
Hugo-DDT force-pushed the fix/bash-tool-bounded-output branch from 28f23be to 5f3ef0f Compare October 22, 2026 12:20
Replace process.communicate() with concurrent 64 KiB chunked reads of
stdout and stderr into per-stream buffers capped by the new
BashToolPolicy.max_output_bytes_per_stream (default 1 MiB, None
disables). Excess output is drained and discarded so the child never
blocks on a full pipe, and the returned text ends with a marker noting
how many bytes were omitted. Timeout handling keeps draining the pipes
after killing the process group so the subprocess is reaped.

Fixes google#7507
Address review on google#7521 / google#7507: on CancelledError kill the process
group, cancel completion and drain readers, gather them, then re-raise;
reject max_output_bytes_per_stream < 0; add exact-at-limit and
cancellation tests.
Drop gather+shield so a setsid descendant holding a pipe cannot leave
process.wait() pending after timeout, and so Python 3.14 does not log
CancelledError from shielded futures. Track readers and an explicit
waiter task, finish all three on timeout/cancel, and cover the setsid
timeout path with a regression test.
@Hugo-DDT
Hugo-DDT force-pushed the fix/bash-tool-bounded-output branch from 5f3ef0f to f3403e4 Compare October 23, 2026 09:04
A setsid descendant survives the process-group kill and keeps the pipe
open, so it keeps running after run_async returns. Close the subprocess
transport once the waiter and readers are finished so that descendant's
next write fails with EPIPE.

Fixes google#7507

Co-authored-by: DDT <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ExecuteBashTool buffers unbounded stdout/stderr in the parent process

3 participants