Skip to content

asyncio: large bytes write is quadratic when queued behind pending data #159503

Closed

Description

@Ar-maan05

Bug report

Bug report

Bug description:

When a bytes object is written to a selector socket transport while data is already buffered, each partial send re-queues the leftover as b[nbytes:] (_adjust_leftover_buffer) or buffer[n:] (_write_send), copying the whole remainder every time. One large write is therefore O(n**2). write() already avoids this on its first-send path with memoryview(data)[n:].

One write queued behind a 2 MiB prefix on a socketpair (Linux, main, best of 3):

size     before     after
16 MiB   205 ms     12.3 ms
48 MiB   1633 ms    14.4 ms
bench_write_copy.py
# Time one large bytes write that has to queue behind data already buffered
# in an asyncio socket transport.
# Usage: python bench_write_copy.py [size in MiB, default 48]
import asyncio
import socket
import sys
import time

SIZE = int(sys.argv[1] if len(sys.argv) > 1 else 48) * 1024 * 1024
PREFIX = 2 * 1024 * 1024


class Reader(asyncio.Protocol):
    """Counts received bytes and resolves `done` once all have arrived."""

    def __init__(self, expected, done):
        self.received = 0
        self.expected = expected
        self.done = done

    def data_received(self, data):
        self.received += len(data)
        if self.received == self.expected:
            self.done.set_result(None)


async def main():
    loop = asyncio.get_running_loop()
    sock_a, sock_b = socket.socketpair()
    done = loop.create_future()
    reader, _ = await loop.connect_accepted_socket(
        lambda: Reader(PREFIX + SIZE, done), sock_b)
    writer, _ = await loop.connect_accepted_socket(asyncio.Protocol, sock_a)

    payload = b"x" * SIZE
    start = time.perf_counter()
    writer.write(b"p" * PREFIX)  # fills the socket, so the next write is buffered
    writer.write(payload)        # sent in many partial chunks
    await done
    elapsed = time.perf_counter() - start

    writer.close()
    reader.close()
    await asyncio.sleep(0)
    print(f"{SIZE >> 20} MiB write: {elapsed * 1000:.1f} ms")


asyncio.run(main())

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

  1. added
    performancePerformance or resource usage
    stdlibStandard Library Python modules in the Lib/ directory
    on Oct 16, 2026
  2. added a commit that references this issue on Oct 29, 2026
  3. moved this from Todo to Done in asyncioon Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytopic-asynciotype-bugAn unexpected behavior, bug, or error

    Projects

    • Status
      Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions