LZ4 Version 1.10: Flotter dank Multithreading
The new version of the LZ4 packing program uses several processor cores, offers a new level of compression and supports dictionary compression.
(Image: rawf8/Shutterstock.com)
LZ4 refers to both a particularly fast compression algorithm and a corresponding reference implementation. The latter consists of the library Liblz4 and the command line program "lz4". The complete system is now available in version v1.10.0, whose code name "Multicores Edition" already indicates one of the biggest innovations.
Multithreading
LZ4 now distributes its work across several threads. On processors with many cores, this leads to a significant speed boost. The jump in performance is particularly noticeable with very high compression levels, which shrink files with a lot of brain power (high compression modes). According to the developers' benchmark results, the new LZ4 version works up to eight times faster than its predecessor. The team recorded the greatest speed increases on a test system with an AMD Ryzen 7 7840HS processor and Windows 11.
In addition, I/O operations and decompression processes overlap. Together with this optimization, LZ4 only needed three instead of five seconds to unpack a 5 GB text file on an unspecified test system. The speed increases not only give the high-compression modes a leg up, but are also useful when using fast storage systems such as NVMe data carriers.
Speaking of compression levels: LZ4 offers a new Compression Level 2, in which the packing program works slightly slower than in the standard Level 1 (Fast Compression), but compresses more intensively. However, the files created are not as small as in the higher level 3. The developers recommend level 2 whenever you need smaller archives than in level 1, but cannot wait as long for them as when using level 3.
Dictionary Compression
The previous version already supported dictionary compression. With this compression method, the packing program replaces recurring file contents with a shorter symbol. The tool remembers the corresponding assignments in a file known as a dictionary. If several files with similar content share a dictionary, this can drastically reduce the storage requirements of the compressed copies. The use of a dictionary is therefore particularly worthwhile for many smaller files that cannot be compressed very well themselves.
While dictionary compression was still considered experimental in the previous version, LZ4 v1.10.0 now officially supports the process. Programmers in particular can finally use dictionaries safely in their applications. The Liblz4 library offers several new functions for this purpose. For example, "LZ4F_decompress_usingDict()" unpacks several data blocks (in the form of a so-called LZ4 frame) with a dictionary. LZ4 can also work with existing dictionaries created by the dictionary builder of the Zstd compression tool.
Finally, LZ4 supports the processor architectures loongArch, RISC-V, m68k, MIPS and Sparc, and the developers have also made slight changes to the build system. All changes are summarized in the official announcement.
(olb)