[andygrove]

What's Changed

Lz4CompressionCodec uses commons-compress's pure-Java LZ4, which is slow enough that compressing an 8 MiB buffer of sequential int64 values takes several minutes (#192). This changes the codec to use lz4-java, the JNI bindings to liblz4 that Spark, Kafka and others use, while keeping every frame it could read before readable.

Compression. Each buffer is written as one LZ4 frame of independent blocks of up to 4 MiB, without checksums, compressed with lz4-java directly from the buffer's memory. Every LZ4 frame decoder reads frames with independent blocks: liblz4 (Arrow C++, pyarrow), lz4_flex (arrow-rs), lz4-java, and commons-compress.

Decompression. A frame with independent blocks is decompressed with lz4-java directly into the output buffer. That covers frames from this codec, from arrow-rs, and from the previous version of this codec. The new decoder accepts any block size, block and content checksums (which it verifies), and a content size. It checks the frame's block lengths, and that they can hold the recorded decompressed length, before allocating for it.

A frame whose blocks are linked, or that uses a dictionary, still goes through commons-compress. lz4-java cannot decode a block that refers to earlier blocks, and Arrow C++ writes linked blocks: liblz4's default LZ4F_preferences_t is LZ4F_blockLinked, so pyarrow 26 writes every buffer larger than 64 KiB as linked 64 KiB blocks. The LZ4 files in arrow-testing are small enough that liblz4 writes them as single, independent blocks, so the integration tests would not catch this. TestLz4CompressionCodec covers linked frames written by commons-compress instead.

Dependency. This adds at.yawk.lz4:lz4-java 1.12.1, the maintained fork of org.lz4:lz4-java. The original artifact's last code release, 1.8.0, has several published advisories (for example CVE-2025-12183 and CVE-2025-66566), and its 1.8.1 is a relocation POM that points to the fork. Spark moved to the fork in 4.2. Both artifacts use the net.jpountz packages and the org.lz4.java module name, and the codec uses only APIs present in both, so an application that pins org.lz4:lz4-java instead still works. The codec asks for lz4-java's JNI implementation and falls back to its Java one only when the native library cannot load, because LZ4Factory.fastestInstance() settles for Java whenever lz4-java is not loaded by the system class loader.

Performance

CompressionCodecBenchmarks (new, in performance) on an Apple M3 Ultra, JDK 17, 8 MiB buffers:

Benchmark	LZ4_FRAME	ZSTD
compress, sequential int64	8.5 ms	12.5 ms
compress, random decimal text	9.9 ms	44.6 ms
decompress, sequential int64	2.1 ms	4.8 ms
decompress, random decimal text	1.9 ms	8.5 ms

The previous codec is too slow for the same JMH run. From a simple timing harness on the same machine:

Input	Previous compress	New compress	Previous decompress	New decompress
8 MiB sequential int64	278 s	7.8 ms	93.5 ms	2.2 ms
8 MiB of i % 1000 as text	17.9 ms	0.3 ms	3.3 ms	0.3 ms
8 MiB random bytes	243 ms	0.5 ms	(stored)	(stored)
1 MiB random decimal text	38.5 s	1.4 ms	33.3 ms	0.3 ms

Compression ratio

On the inputs above, the ratio is unchanged except for random decimal text, where commons-compress's more thorough match search reached 1.61 and lz4-java's fast compressor reaches 1.10. The new output matches what Arrow C++ writes by default: pyarrow's LZ4 codec also reaches 1.10 on the same kind of data, and 1.63 at compression level 9. CommonsCompressionFactory.createCodec(LZ4_FRAME, level) ignores the level today. Mapping it to lz4-java's high-compression mode would give users who want the better ratio a way to ask for it; I left that out to keep this change focused, and can add it here or in a follow-up.

Testing

• The existing TestCompressionCodec and TestArrowReaderWriterWithCompression pass unchanged, including the decompressed-length check from GH-1206: validate decompressed length in Lz4CompressionCodec #1207.
• TestLz4CompressionCodec (new):
  • round-trips buffers across block boundaries, including one whose first block is incompressible and stored, and checks that commons-compress and lz4-java's LZ4FrameInputStream both read the frames this codec writes;
  • decompresses frames written by commons-compress for every block size, with linked and independent blocks, with and without checksums, and a frame written by lz4-java with a content size;
  • rejects corrupt frames (descriptor, block and content checksums, a missing end mark, blocks that run past the frame or exceed the maximum block size, undecodable blocks, a wrong content size, and recorded lengths that are too long or too short), and releases everything it allocated.
• I also decoded pyarrow 26 output (buffers over 64 KiB, linked blocks) by hand to confirm that lz4-java rejects linked frames and commons-compress reads them, which is why the fallback stays.

AI usage

The code, tests and benchmark in this PR were written with an AI coding assistant. The frame behavior described above (pyarrow's linked blocks, lz4-java's rejection of them) and the benchmark numbers come from running them.

Closes #192.

[jbonofre]

The CI is failing on Spotless check on Lz4ComporessionCode.java.

[jbonofre]

arrow-tools shades every runtime dependency, so the release binary distribution will include lz4-java and the native libraries it ships, which are built from LZ4 and xxHash (BSD-2 Clause).

We have to add entries to tools/src/shade/LICENSE.txt for lz4-java 1.12.1 (Apache License 2.0) and for LZ4 and xxHash (BSD-2 Clause, with the license content inline).

It's a blocker for release.

[andygrove]

Added in 2f3abec: lz4-java 1.12.1 (Apache 2.0), plus LZ4 1.10.0 and xxHash 0.6.5 with their BSD 2-Clause texts inline. lz4-java 1.12.1 pins LZ4 at v1.10.0 and builds its native libraries from lz4.c, lz4hc.c and xxhash.c in LZ4's lib directory, so the xxHash it ships is the copy there. lz4-java has no NOTICE file, so NOTICE.txt is unchanged. The shaded arrow-tools jar now contains the nine lz4-java native libraries and the three entries. The spotless violations are fixed in 9a2cd0b.