After 20 years of painstaking work: Linux kernel now real-time capable

Contents

After two decades and tens of thousands of changes everywhere, a group of developers has finally achieved its goal: as of Friday this week, the official Linux kernel has everything it needs to fulfill hard real-time requirements. In other words, even under adverse environmental conditions, it is guaranteed to carry out a predefined task within the specified time. This is essential for many robots in industrial production or self-driving vehicles, for example, but it is also beneficial for the low-latency and smooth processing of audio and video as well as numerous other areas of application. For decades, it was generally said in research that it was impossible to realize hard real-time with a general-purpose operating system such as Linux.

Users will achieve real-time capabilities on November 18 or 25 with Linux 6.12, unless something unforeseen happens between now and then. The standard kernels of common Linux distributions will certainly still not offer hard real-time support. Moreover, the milestone that has now been reached does not mean that Linux is now teasing hard real-time capabilities out of any hardware and software. The work of the developers who have been working on the PREEMPT_RT patch and other important changes for real-time support in the RT tree for years is by no means complete. But first things first.

It got tough again in the last few meters

Linux should be able to guarantee response times of 100 microseconds thanks to the new real-time support, according to developers – assuming ARM64, RISC-V or x86-64 platforms that are common and suitable for real-time use. On average, even 10 microseconds are possible.

The last major change required for this was the final modifications to the Printk infrastructure, which were adopted for Linux 6.12 on Tuesday. The kernel uses the latter to accept the log messages that can be retrieved via dmesg, for example, and then outputs them via virtual or serial consoles.

If there is a problem with the kernel, this log is often the essential main source of information. Synchronization is extremely important in order not to mix up incoming messages into a mess. Until now, however, the Printk code has tended to revolve around itself, which has delayed the execution of other, but more important, programs running in parallel. This is exactly what must not happen with a self-propelled warehouse robot, for example, because otherwise it might take a turn a little too late and knock over a shelf in the process.

The conversions now integrated by John Ogness and Thomas Gleixner represent the second attempt to eliminate the Printk weakness. The first made it into the kernel in the summer of 2022, after the developers had been working on the whole thing for over three years instead of a few months. However, bugs in the synchronization of messages emerged shortly afterwards. As a result, the developers kicked the whole thing before the completion of Linux 5.19 to tackle the matter differently – and that just took another two years. This happens very rarely, but is now once again within the realm of possibility, so it is still possible that a backpedaling before the release of 6.12 is possible.

Real-time core already integrated over two years ago

The Printk conversion was followed by three comparatively small changes with a major impact at 6 a.m. on Friday: They mean that the CONFIG_PREEMPT_RT option can now be activated when configuring to build a kernel for ARM64, RISC-V and x86-64 systems (32- and 64-bit), which leads to the building of a "Fully Preemptible Kernel (Real-Time)".

Until now, this option was not selectable, although the code responsible for it was already included in Linux 5.15 with the core of the real-time patches. The latter means that a kernel compiled with CONFIG_PREEMPT_RT handles central locking techniques such as spinlocks, mutexes and rwlocks differently so that it can interrupt almost all the tasks it performs without major delays.

As a result, Linux can turn to important tasks extremely quickly at almost any time: One or more programs marked with real-time priorities, which must always and absolutely complete their task within a predefined period of time. In other words, even if an insignificant maintenance process is currently placing immense demands on the system and creating very adverse circumstances –, such as a file system check, an update tool or log analysis software. The real-time prioritization of the process determines which programs the kernel should prioritize and to what extent. If desired, the time allocation can be controlled even more specifically with the deadline scheduler, which was introduced in kernel 3.14 in 2014 and was specially designed for real-time requirements.

Real-time support for distributions usually an extra sausage

With real-time support in the official kernel, more distributors are likely to be willing to deliver a real-time-capable kernel in addition to the standard kernel. Separate kernel images including suitable modules are currently common, as the greater willingness to switch contexts can slightly reduce disk or network throughput, for example – in the same way that people sometimes need longer to complete a task, interrupting this activity briefly for each parallel incoming email, chat or social media message instead of occasionally processing them in a bundle.

Approaches such as "lazy preemption" promise to avoid this disadvantage completely or to a large extent if real-time is not required. However, it is still uncertain whether the modifications required for this will ever mature to the point where they are incorporated into the official kernel; if they do make the leap at some point, mainstream distributions could abandon the special real-time kernel images that cause extra work.

20 years of development work

The PREEMPT_RT developers in the RT tree have driven forward all of the changes to real-time support that have been incorporated now and most of those before, and brought them to integration maturity; at the same time, they have also created versions that can be used by interested parties and maintained them for years. With this kernel branch and the PREEMPT_RT patch created shortly afterwards, the development of real-time support for the official Linux kernel really began shortly after the start of fall 2004.

In view of the twentieth anniversary of these efforts, a small party was held on Thursday in the context of the Linux Plumbers Conference (LPC) currently being held in Vienna. After the Printk changes had been incorporated a few days ago, Thomas Gleixner handed over the request to include the three above-mentioned changes for PREEMPT_RT support to Torvalds personally and in printed form, bowing for the help; normally the Linux inventor and main developer receives such by mail without such fuss.

Gleixner from the German company Linutronix is the main driving force behind real-time support for the standard kernel. However, he and his many helpers can by no means simply retire now, as there is still a lot of fine-tuning to be done. For example, some changes will probably not make it into kernel 6.12 so that the driver for serial interfaces does not linger too long in uninterruptible code when logging via serial consoles.

In any case, the developers of the RT tree will have to be constantly vigilant in the future to make sure that any changes made by any kernel developer do not unintentionally trigger long latencies: such a faux pas in critical code paths can happen quickly. And one is enough to ruin the real-time suitability of Linux.

RT support for further processor architectures

Support for the PowerPC architecture and 32-bit ARM systems is also still under discussion, but still requires minor adjustments. In addition, there are still some kernel parts that are known to be prone to errors, such as support for the transparent use of large memory pages (Transparent Huge Pages) or the i915 graphics driver responsible for many Intel graphics cores. An LWN.net article from 2023 on a Q&A session with Gleixner provides some background information. There are probably numerous other undiscovered problems lurking in drivers –, especially if no one has ever used them in real-time operation.

In any case, hard real-time is not something that can simply be activated with a handshake – Instead, users often have to individually optimize and test their hardware, software and kernel to ensure that a real-time-capable operating system actually achieves the desired response times for common real-time applications in practice.

John Ogness recently summarized a rough impression of the pitfalls in a blog post. Power-saving technologies, for example, tend to shoot across the line and therefore often need to be deactivated. According to Ogness, some hardware is also completely unsuitable. This can be due to system management interrupts (SMIs), for example, where the motherboard firmware temporarily takes control away from Linux in order to take care of the emulation of outdated technologies, fan control, management engine and similar things –, which quickly leads to unintentionally high latencies that the operating system can only watch powerlessly.

General optimizations by the RT developers

For most notebooks, desktops and servers, the new real-time capabilities of the kernel are therefore ultimately not particularly relevant. Nevertheless, practically all Linux users have a lot to thank the developers of the RT tree for: they have made thousands of modifications that have significantly improved the kernel in numerous areas over the last two decades. For example, the real-time developers noticed and eliminated many scaling problems long before they became a problem on common hardware –, which is one of the main reasons why Linux was already good at exploiting the potential of systems built with multi-core processors when they first appeared.

Now that it has finally been successfully included in the main development branch of Linux, it should attract new interested parties. With any luck, some of them will also get involved in maintenance and further development. This has been on very shaky ground several times over the past two decades; at times it even came to a standstill because many companies using the RT tree did not get involved at all or only minimally. In addition to the complexity of the whole thing, this is one of the main reasons why development has taken almost two decades to reach its current state. Congratulations to all the developers involved and thank you for your commitment!

(mki)