You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Optimize LittleEndianDataOutputStream.writeInt/writeShort with bulk write (resolves existing TODO) #3518
When the underlying stream is CapacityByteArrayOutputStream (the typical case in Parquet writers), each out.write(int) performs a hasRemaining check, a Math.addExact for the new size, possibly a slab-grow check, and a single-byte store. For writeInt, that's 4 separate trips through the bookkeeping.
The class already has the right pattern in writeLong: build a writeBuffer[] and emit a single out.write(writeBuffer, 0, 8). The buffer is even pre-allocated for that purpose. writeInt and writeShort just don't use it.
There's a TODO comment in writeInt (lines 147–149 in current master) acknowledging this:
// TODO: see note in LittleEndianDataInputStream: maybe faster
// to use Integer.reverseBytes() and then writeInt, or a ByteBuffer
// approach
Proposal
Extend the existing writeBuffer[] pattern to writeInt and writeShort:
publicfinalvoidwriteInt(intv) throwsIOException {
writeBuffer[0] = (byte) (v >>> 0);
writeBuffer[1] = (byte) (v >>> 8);
writeBuffer[2] = (byte) (v >>> 16);
writeBuffer[3] = (byte) (v >>> 24);
out.write(writeBuffer, 0, 4);
}
This collapses 4 write(int) calls into 1 write(byte[], int, int) call, cutting the bookkeeping overhead by ~4x per int. Matches the existing writeLong pattern in the same file.
Resolves the existing TODO in the source.
Expected impact
Standalone JMH benchmark of the class:
IntEncodingBenchmark.encodePlain (when routed through LittleEndianDataOutputStream): ~+35% (~20.9M → ~28.2M ops/s)
Note on context
PR #3496 deprecates LittleEndianDataOutputStream because Parquet's own writers no longer use it (they write directly into ByteBuffer-backed slabs, which compiles to a single intrinsic store on little-endian architectures and is strictly faster than any wrapper).
This PR is therefore a purely external-caller benefit: any third-party Parquet-format producer still using the class will get the speedup until they migrate. The change is minimal (~10 lines), obviously correct (matches the existing writeLong pattern), and resolves a long-standing TODO in the source.
Activity
iemejia commentedon Apr 21, 2026
Folded into #3495 / PR #3496.