[ivonastojanovic]

Tachyon's binary format currently stores all samples in a single compressed stream. Each record contains its thread and interpreter IDs, timestamp, status, and stack change, so values such as the thread and interpreter IDs are written repeatedly across records. The existing RLE encoding avoids some of this when a stack repeats, and compression reduces it further, but the format still mixes fields with very different patterns in the same stream.

More importantly, this means replay has to decompress the complete sample stream before it can process the first sample. The temporary buffer therefore grows with the length of the capture, even though replay only needs a small part of the sample data at a time.

This PR changes the binary format to split samples into bounded, independently decodable blocks. Replay can decompress and process one block at a time, then reuse the same buffers for the next block. Within each block, sample fields are also separated into columns, allowing repeated values to be removed before compression and each type of data to use an encoding that better matches its structure.

This change focuses on profiles exported to regular files. Live streaming remains separate work in #145464. Regular files can keep shared string and frame tables at the end of the file, while a live stream cannot refer to tables that have not been sent yet. For streaming, we may instead need to include the required string and frame data in each block so that it can be decoded as soon as it arrives.

Implementation

The new file layout is:

header | sample block | sample block | ... | string table | frame table | stats | footer

Each sample block has its own header followed by six columns:

block header
    | stream table
    | stream IDs
    | timestamp residuals
    | status bitmaps
    | stack instructions
    | frame IDs

Samples keep their original order across these columns, but no longer need a complete record containing every field. Values shared across a block move into its header or tables, while the per-sample columns only store what changes. The block header contains the sample count, base timestamp, frame-ID width, status masks, and the size and encoding of each column.

Column	What it stores	What this avoids
Stream table	Each (thread_id, interpreter_id) pair used by the block, once	Repeating both full IDs in every record
Stream IDs	A small stream-table index for each sample	Carrying the full thread and interpreter IDs with each sample
Timestamp residuals	Difference from the expected timestamp for that stream	Storing larger timestamp differences alongside other fields
Status bitmaps	One bitmap for each status bit that varies	Writing one complete status byte per sample; common bits are stored once in the header
Stack instructions	How to repeat, change, or reuse a stack	Writing frames that are unchanged or already available in the block
Frame IDs	Referenced frame-table indices at the smallest width needed by the block	Using a larger representation for every frame ID

The writer first reserves space for the file header. It then collects samples into a block and starts a new one when the decoded column data reaches 1 MiB or the block reaches 4,096 unique (thread_id, interpreter_id) pairs. Once sampling finishes, it writes the shared string and frame tables, statistics, and footer, then seeks back to fill in the file header.

Strings and frames remain shared across the whole file, preserving deduplication across the complete capture. These tables still grow with the number of unique strings and frames, but temporary decoded sample data is now bounded by the size of one block rather than the total number of samples.

Timestamp and stack state are tracked independently for each stream. Timestamps are stored as residuals from the expected timestamp, so samples arriving at the normal interval usually store zero. Status bits shared by all samples in a block are stored once in the block header, while only varying bits need per-sample bitmaps.

Stacks use three instructions:

Instruction	Operands	Meaning
REPEAT	count	Reuse this stream's previous stack for count samples
REPLACE_SUFFIX	keep_count, new_count	Keep the common outer frames and append new_count frame IDs
REUSE	offset, count	Copy a complete stack already stored in the block's frame-ID column

REPEAT is tracked per stream, so samples from other streams can appear between repetitions. The other instructions either write only the frames that changed or reuse a complete stack already stored in the block. Frame IDs use between one and four bytes and are stored in byte planes to improve compression.

Each column is stored as raw bytes, a single repeated byte, or zstd level 3, whichever is smallest. The file-wide string and frame tables similarly choose between raw and zstd.

After discussing this with @pablogsal, we decided not to keep support for version 1 profiles since this change will be shipped in 3.16.

The public BinaryWriter, BinaryReader, and collector APIs remain unchanged, except that the get_stats() dictionaries now report reuse_records instead of pop_push_records, since the pop-push encoding no longer exists.

• Issue: Tachyon: use bounded columnar blocks for binary profiles #159207

[ivonastojanovic]

cc @maurycy

[ivonastojanovic]

I did some benchmarking while working on this, so leaving the results here for reference.

For the main comparison, I profiled test_list, test_dict, test_tokenize, test_exceptions, test_syntax, and test_threading with non-debug builds on an Apple Silicon Mac. The number of samples was slightly different between runs, so I compared bytes per sample instead of the final file size.

Compression	main	This PR	With compressed tables
None	2.4121	1.6643 (-31.0%)	N/A
zstd	0.3186	0.2787 (-12.5%)	0.2572 (-19.3%)

I also experimented with compressing the file-wide string and frame tables, so I included those numbers in the last column. It helps a bit more with zstd, but I left it out of this PR since the change is already pretty large. We can always add that separately.

I also tried two smaller programs with zstd to see how the format behaves with different workloads. One does JSON and zlib work across eight threads, so repeated stacks are interleaved across several streams. The other runs SQLite queries and processes the results in Python, so it mostly stays on one stream and moves between a small set of stacks.

Workload	main	This PR	With compressed tables
JSON/zlib threads	0.4659	0.2508 (-46.2%)	0.2652 (-43.1%)
SQLite queries	1.3747	1.0991 (-20.0%)	1.0896 (-20.7%)

The small differences between the two block-format results are just normal run-to-run variation. These are also bytes per sample.

I also checked a few synthetic stack patterns with zstd: one stable stack, stacks where only the inner frames change, complete stacks shared across threads, and a bank of randomly selected stacks.

Workload	main	This PR	With compressed tables
Stable stacks	1.6 KiB	1.4 KiB (-13.9%)	1.2 KiB (-28.6%)
Common outer frames	323.1 KiB	129.1 KiB (-60.0%)	108.7 KiB (-66.4%)
Shared complete stacks	518.2 KiB	125.2 KiB (-75.8%)	50.8 KiB (-90.2%)
Random stack bank	734.6 KiB	3,506.1 KiB (+377.3%)	3,385.9 KiB (+360.9%)

The random stack case is the obvious regression. The frame IDs do not compress particularly well, and repeated stacks can end up in different blocks, so we lose the ability to reuse them across block boundaries.

The main thing I wanted to improve with this change was replay memory, though. With the previous format, peak memory grew from 31.0 MiB at 100,000 samples to 59.4 MiB at one million samples. With the block reader, it stayed at around 26.3 MiB.

[ivonastojanovic]

I also tried with just REPEAT, but REPEAT, count is smaller and generally faster, especially without compression

[ivonastojanovic]

Before we used one byte per sample, so for 1000 samples the old format uses 1000 bytes. With bitmaps, one varying flag only needs 127 bytes. Even in the worst case, where all eight flags vary, we need 1002 bytes, so it's basically the same size as before.

[ivonastojanovic]

I also experimented with larger 8 MiB blocks, but overall 1 MiB worked better. The 8 MiB blocks only helped meaningfully in the synthetic case with repetition across many samples, since more stack history stayed within the same block.

[maurycy]

@ivonastojanovic Thank you! This is the top of my queue. Two extremely quick notes.

[pablogsal]

I took a first look at the PR. A few ideas for this change or later PRs:

• Add a small, bounded cache in the reader that maps frame IDs to immutable FrameInfo objects. Reuse these objects across samples, while each callback still receives a fresh frame list. I suspect this will reduce the CPU spent on repeated object allocation.
• For long captures, avoid a whole-file preload. On Linux, remove eager page population and release pages after the reader finishes with them. On Windows, replace the whole-file buffer with reads of individual blocks. This looks useful for keeping resident memory low as captures grow.
• Give each hash bucket in the writer cache room for two or four stacks. Currently, stacks that map to the same bucket replace each other, even if both recur. Keep the total number of slots unchanged. I expect fewer collisions to discard useful entries.
• The reuse cache also excludes stacks encoded as changes to a previous stack. For a repeated stack, consider a full-stack record so later occurrences can use REUSE. That gives us another opportunity for compression, provided the later savings cover the larger initial record.
• Base the block cutoff on decoded column size, rather than the larger temporary buffers. Keep a separate memory limit for those buffers. I think this will give us fuller blocks and less overhead per sample. The trade-off to check is whether larger flushes cause longer pauses.

[ivonastojanovic]

I took a first look at the PR. A few ideas for this change or later PRs:

• Add a small, bounded cache in the reader that maps frame IDs to immutable FrameInfo objects. Reuse these objects across samples, while each callback still receives a fresh frame list. I suspect this will reduce the CPU spent on repeated object allocation.

Done. On a two-million-sample repeated-frame profile, median replay time dropped from 0.98s to 0.83s (about 15%), while CPU time dropped by about 23%.

• For long captures, avoid a whole-file preload. On Linux, remove eager page population and release pages after the reader finishes with them. On Windows, replace the whole-file buffer with reads of individual blocks. This looks useful for keeping resident memory low as captures grow.

I'll handle this in a follow-up PR. I don't have a good way to test on Windows right now, so I'll open one next week.

• Give each hash bucket in the writer cache room for two or four stacks. Currently, stacks that map to the same bucket replace each other, even if both recur. Keep the total number of slots unchanged. I expect fewer collisions to discard useful entries.

Done.

• The reuse cache also excludes stacks encoded as changes to a previous stack. For a repeated stack, consider a full-stack record so later occurrences can use REUSE. That gives us another opportunity for compression, provided the later savings cover the larger initial record.

I'll experiment with this in a follow-up as well, since this PR is already getting pretty large.

• Base the block cutoff on decoded column size, rather than the larger temporary buffers. Keep a separate memory limit for those buffers. I think this will give us fuller blocks and less overhead per sample. The trade-off to check is whether larger flushes cause longer pauses.
• rather than the larger temporary buffers

Follow up.

So @pablogsal, I would leave this as it is for now and experiment with a few things in follow-ups:

• Points 2, 4, and 5 that you mentioned here
• Compressing the string and frame tables, or potentially compressing the whole file at the end

Let me know if you think any of these should be fixed in the first round.

The latest writer-cache change mainly improved shared complete stacks, reducing this PR's result from 125.2 KiB in the earlier run to 82.8 KiB; the other synthetic results stayed roughly the same.

Compression	main (bytes/sample)	This PR (bytes/sample)
None	2.4121	1.6687 (-30.8%)
zstd	0.3186	0.2986 (-6.3%)

Workload	main (bytes/sample)	This PR (bytes/sample)
JSON/zlib threads	0.4659	0.2700 (-42.0%)
SQLite queries	1.3747	1.0715 (-22.1%)

Synthetic stack pattern	main	This PR
Stable stacks	1.6 KiB	1.4 KiB (-13.9%)
Common outer frames	323.1 KiB	128.6 KiB (-60.2%)
Shared complete stacks	518.2 KiB	82.8 KiB (-84.0%)
Random stack bank	734.6 KiB	3,506.1 KiB (+377.3%)

[maurycy]

@ivonastojanovic I started writing lots of inline comments but I realized it's getting very long, so following Pablo's format:

• I'd check if buffering the first sample has implications for some formats. I'm thinking of GeckoCollector here in particular. The tests should check this, too!
• Perhaps choose_stack_encoding() can be gotten rid of. That's just a matter of style, though.
• There's a wider discussion about counts vs. timestamps. See the trick from gh-149718: Aggregate same stack frames in Tachyon in some collectors #149719 but in the binary format. I think of pstats, collected and flamegraph here. Two notes: this can combine non-consensutive occurences, and maybe REUSE could identify these before even create frames. That's a general question about decode_block(): how lazy decoding (materialization) can be?
• Similarly: https://github.com/python/cpython/pull/160313/changes#diff-dfdcb94821b5902e4d751f5d78e385c19c1efa6ef2ed808e358741e41fda369bR1063-R1067 maybe there's no need to decode / memset?
• Maybe in build_frame_list() we can read in the reverse order, without temporary stacks in flush_stream_batch().
• Each stream retains a prev_stack buffer until writer destruction. Each buffer reserves frame IDs. When a stream enters a new block, block_add_stream() resets its stack depth and timestamp. The previous block's stack contents are no longer needed. Maybe we can we keep the ID registry global, but keep stack, timestamp, and repeat state only for streams in the current block? This also avoids reset growing linearly.
• Thought experiment: the stream count gives us the bit width needed for the indices. With four streams, we only need two bits instead of a byte.
• Perhaps we can compare independent zstd blocks with the columnar format. Then try REUSE, repeat counts and the bit/byte planes separately. I'm curious how much comes from splitting the data into blocks, how much from each encoding. File size, write/replay CPU time, peak memory, distribution of flush times etc. I can do this!

Thank you! I promise more!