Linux 6.9 with Virtual Data Optimizer released

Contents

Linus Torvalds released Linux 6.9 in the night from Sunday to Monday. With VDO, the latest incarnation of the kernel promises to save disk space and improve the performance of FUSE file systems. In addition, the kernel team is systematically removing old, obsolete and no longer usable file systems.

Deduplication with VDO

The "Virtual Data Optimizer" (VDO) is a device mapper. As such, it inserts itself between the actual storage medium and the file system. It thus offers a virtual block device in the direction of the file system, which can address it like a hard disk, SSD or SAN. The virtual devices from VDO are therefore similar to the virtual volumes of LVM (Logical Volume Manager).

VDO promises to reduce storage space requirements through "deduplication". Instead of writing the same data multiple times to the storage medium, it recognizes duplicates of blocks or data records and combines them. To do this, it creates virtual block addresses that are passed on to the storage users. Internally, it bundles these into a physical block (shared block) on the storage device.

As soon as a write access is made to this shared block, VDO creates a new physical block. The write virtual block address then points to the new physical block. All other virtual blocks remain on the old physical block. A classic "copy on write", but not as a file system feature, but as a device mapper.

Compressed with VDO

In addition to deduplication, VDO also offers the option of compressing the underlying storage medium. It always compresses blocks that are not shared blocks, i.e. only exist once. When a new physical block is created, VDO compresses it by default. The LZ4 algorithm is used for compression.

Compression works best with file formats that do not offer their own redundancy at block level. For file system logs and physical volumes of databases, VDO compression is rather inconvenient or even useless. For "normal" data on file systems, on the other hand, compression helps to reduce storage requirements in addition to deduplication.

No chance for zeros

Thanks to "zero block elimination", VDO also only manages real data in blocks. All blocks that only contain zeros are not really stored by VDO. Zero blocks do not occupy any memory.

VDO was already in use outside the mainline kernel on production systems. Permabit Technology originally developed VDO in 2009 and released it for production systems in 2013. When Red Hat acquired Permabit in 2013, the originally proprietary software became open source and found a permanent place in Red Hat's operating systems, above all the enterprise distribution RHEL. With Linux 6.9, the productively tested VDO is now an integral part of the mainline kernel.

Old NTFS is passé

The new kernel deletes the old ntfs driver in favor of the more powerful ntfs3. Linux 5.15 introduced the "ntfs3" driver from Paragon for the Windows file system. This supports NTFS 3.1 and is therefore much more compatible with Windows than the old, almost forgotten driver.

The old ntfs driver lacked many functions. In addition, its rudimentary write access could lead to one or two unpleasant surprises. Most distributions have therefore already switched to the FUSE driver (Filesystem in Userspace) ntfs-3g. To ensure that mounts with nfts as the file system continue to work, with Linux 6.9 ntfs is simply an alias to ntfs3. Anyone who still mounts NTFS file systems with nfts, for example via an entry in /etc/fstab, does not have to change anything initially. With the kernel change, ntfs3 automatically serves the file system. This alias is comparable to ext3, which is also an alias to ext4.

ext2 is obsolete

The veteran among the file systems, ext2, is marked as "deprecated" (obsolete) by the new kernel. This step is necessary because ext2 cannot handle data beyond January 19, 2038. The 32-bit based Unix time will then overflow and start again with January 1, 1970 without any measures. The ext2 driver can still be used. However, the kernel acknowledges its use with a warning indicating the date problem. Alternatively, ext2 file systems can also be mounted with the ext4 driver.

Ideally, however, the use of ext2 file systems should always be avoided. At least that is the recommendation, which is likely to generate additional work for systems that have been consistently updated over the years. ext2 was often used as the last remnant for /boot. After all, system administrators of such systems still have enough time until 2038. ARM64 systems are an exception. There, the kernel team is deleting ext2 and the redundant ext3 completely. Anyone using ext2 on such systems will have to mount them with the ext4 driver in future. ext3 does not require any manual intervention, thanks to the above-mentioned alias to ext4.

High-performance FUSE

FUSE (Filesystem in Userspace) allows file systems to be implemented in userspace instead of as a kernel module. A server located in userspace maps the actual file system with its file system structures and metadata. This server uses the FUSE framework provided by Linux for the necessary privileged hardware access.

Until now, all FUSE accesses to and from the kernel have been routed through the corresponding server. In principle, this reduces performance and may be disruptive in some applications. Linux 6.9 now has the first "FUSE passthrough" for this. This allows I/O accesses – to be made directly to the kernel as soon as the corresponding file is opened –. The server no longer acts as a "pass-through heater".

The respective FUSE file system or its server must support the new feature. Furthermore, this only works with privileged servers, i.e. those that have the CAP_SYS_ADMIN capability set. Later versions of FUSE-Passthrough should also work without this privilege.

With NFSv4, system administrators have the option of resetting the open and lock status of files. Work on the XFS journaling file system is progressing to enable a file system check and repair when the file system is mounted (online check and repair). However, this is not yet productively possible in Linux 6.9.

Processor architectures

Rust can now also be used on 64-bit ARM systems for writing kernel modules. The LPA2 mode for setting 52 bits for virtual addresses has also arrived in the kernel.

A kernel can now also be patched live on the Chinese LoongArch architecture. The"ORC Stack Unwinder" is also available on LoongArch.

On IBM's S/390 mainframe architecture, the kernel had long equated the virtual address with the physical address. In fact, the values were identical. Rather unusual for the mainframe environment, this led to considerable inaccuracy. Whether the value of the virtual or the physical address was used, it worked even when they were mixed up. There are now efforts to separate the kernel space from the user space on S/390. This can only work by keeping virtual and physical addresses cleanly separate. The new kernel therefore introduces some new types to mark physical addresses. The aim is to use tools to identify incorrectly used addresses and the corresponding code.

To summarize

Linux 6.9 is a consistent maintenance release that primarily carries out clean-up work. VDO is new, but mainly outside the Red Hat world. The FUSE passthrough gives hope for more performance and throughput in the userspace file systems.

The new Linux kernel is available for download at kernel.org. All changes can be found in the detailed kernel changelog.

(dmk)