ResponseNotRead and aiter_bytes

[a14n]

The following code open a stream and iter on all chunks.

async with async_client.stream("GET", url) as response:
    status_code = r.status_code
    async for chunk in response.aiter_bytes(chunk_size=max_response_size_b):
        pass
    response.content # should be OK but raise a ResponseNotRead

Although all chunks have been consumed response.content remains unusable (by raising a ResponseNotRead).

It would be great to set response._content once the iteration is over. WDYT?

[karpetrosyan]

It's the point of streaming, to not load the entire data into the memory, and if it will not raise an error, what should it return?

[a14n]

ok, I didn't understand the point of streaming.
The initial issue I faced was to ensure the content size was not too big and prevent out of memory. So I stream the response. If the content is too big I raise an error but if the content is small enough I'd like to return a response object where response.content and other are usable (without ResponseNotRead). Today I cheat by setting response._content but it's not clean.
Is there a way to control the content size?

EDIT: BTW with the stream you can call await response.aread() and have response.content usable without ResponseNotRead. So even if it is streaming the entire data ends up in memory.

[ketpatil77]

For the size-limit use case in your follow-up, explicitly collect a bounded body, then create a new buffered Response through the public constructor. Iterating a stream normally discards chunks after you consume them, so completing the iterator does not imply that HTTPX retained a copy for .content.

import httpx


class BodyTooLarge(ValueError):
    pass


async def get_buffered(client, url, max_bytes=1_000_000):
    if max_bytes < 0:
        raise ValueError("max_bytes must be nonnegative")
    async with client.stream("GET", url) as response:
        response.raise_for_status()
        body = bytearray()
        async for chunk in response.aiter_bytes(chunk_size=64 * 1024):
            if len(body) + len(chunk) > max_bytes:
                raise BodyTooLarge(f"Decoded body exceeds {max_bytes} bytes")
            body.extend(chunk)

        headers = response.headers.copy()
        for name in ("content-encoding", "content-length", "transfer-encoding"):
            if name in headers:
                del headers[name]
        headers["content-length"] = str(len(body))
        return httpx.Response(
            response.status_code,
            headers=headers,
            content=bytes(body),
            request=response.request,
        )

The returned object's .content, .text and .json() are available normally, without assigning _content. It is a new buffered response, not the original object's complete history/timing state.

Two details matter here:

• Count bytes actually yielded, rather than trusting Content-Length. aiter_bytes() yields decoded data, so the limit also applies to the decoded result of gzip content.
• Since the new body is decoded, remove the original encoding/length/framing headers. Otherwise a new Response could try decoding it again, or expose a length for the compressed representation. Set the length of the buffered body explicitly.

The streaming context closes on success, an oversized body, or a read error. This bounds the body retained by application code, not the entire process RSS: HTTPX decoding/chunk buffers and the final bytes copy still consume memory. Calling aread() directly would buffer the whole response before you could enforce this policy.

Async streaming documentation. AI-assisted; checked small/empty bodies, oversize rejection, gzip header normalization and stream cleanup on HTTPX 0.28.1.