Tag Archives: Phoronix Test Suite

Linux 5.15 Addressing Scalability Issue That Caused Huge IBM Servers 30+ Minutes To Boot


LINUX KERNEL --

Very large IBM mainframes/servers were taking 30+ minutes to boot the Linux kernel… No, just not for POST’ing the system with memory training and the like, but for loading Linux. Fortunately, with the Linux 5.15 kernel there is a set of scalability enhancements to allow these large IBM systems to be able to boot in around five minutes.

With the driver core changes for Linux 5.15 is a set of patches working on enhancing the performance of Kernfs for functionality used around pseudo file-systems like sysfs. Leading to this Kernfs locking and concurrency improvements were engineers finding that large IBM Power mainframe systems with “several hundred CPUs and 64TB of RAM” were taking 30+ minutes to just boot the Linux kernel. Extra kernel parameters were also needed to avoid the kernel timing out on boot.

The extremely long boot times for modern, high-end servers was found to be the result of many path look-ups for non-existent files and extreme locking contention within the VFS code.

Making matters worse is that with the 64TB of system memory and IBM Power dividing them into 256MB local blocks exposed via sysfs, a heck of a lot of sysfs nodes are being created.

With the Kernfs scalability improvements for benefiting sysfs that are in the driver core changes for Linux 5.15, these IBM systems can go from 30+ minutes to boot to now under five minutes. The changes involve switching Kernfs mutex to using a read-write semaphore for allowing node searches in parallel, improving path resolution, and using the VFS negative dentry caching.

These Kernfs improvements and more can be found via the driver core PR that was merged today for the 5.15 merge window.


BPF Timers To Intel Additions Lead The Networking Changes With Linux 5.15


LINUX NETWORKING --

The networking subsystem updates for the recently opened Linux 5.15 merge window have landed.

Among the work exciting me with the networking changes in Linux 5.15 includes:

– Support for BPF timers along with a variety of other BPF enhancements.

– Initial support for Intel’s unreleased “Bz” WiFi hardware to IWLWIFI. The Intel IWLWIFI driver also now supports scanning of hidden 6GHz networks.

– On the Intel wired side is initial support for Lunar Lake with their e1000e driver.

– The Intel Gigabit Ethernet driver IGC also now supports Credit-Based Shaper (CBS) offloading for hardware traffic prioritization and bandwidth reservation. This IGC driver support works with Intel hardware at least of the i225 controller.

– Support for MCTP as the Management Component Transport Protocol. The Management Component Transport Protocol is defined by the Distributed Management Task Force for communicating between relevant controllers and their devices. MCTP can work with buses such as PCI Express, USB, I2C, SMBus, and others while being relevant not only for servers but also embedded devices. Intel NICs have been among the devices supporting MCTP over PCI Express now for a number of years.

– The addition of the LiteETH network driver as part of the LiteX work for FPGA cores.

– The Renesas “RAVB” driver gains support for Gigabit Ethernet IP.

– More scalable and reliable Open vSwitch.

– The Xen netfront code has been hardened against malicious back-ends.

A full list of the networking feature changes for Linux 5.15 can be found via this pull request that as of a few hours ago was successfully merged to mainline.


Con Kolivas Contemplates Ending Kernel Development, Retiring MuQSS & -ck Patches


LINUX KERNEL --

Con Kolivas has worked on many patches for the Linux kernel over the past two decades and particularly focused on innovations around desktop performance/interactivity. For over a decade now he’s primarily been focused on maintaining his work out-of-tree and not catering to mainline acceptance but now he is thinking of bowing out once more and ending his kernel development effort.

Over the past decade he’s been maintaining his “-ck” patches out-of-tree and updating them for each new kernel series with a variety of improvements to enhance the interactivity and performance of the kernel. He’s also been maintaining his MuQSS scheduler that is the successor to his former “BFS” Brain Fuck Scheduler.

While Con’s work in recent times hasn’t been mainlined, these patches have been carried by some distribution kernels and various third-party kernel builds like Liquorix and friends.

With Con Kolivas being a anaesthetist by profession and just a kernel hacker as a devoted hobbyist, last year he took a break from kernel development to design COVID-19 equipment during the early days of the pandemic. However, now he’s looking at taking a longer break or potentially a permanent departure from Linux kernel work.

Con didn’t get around to updating MuQSS and his -ck patches for Linux 5.13 and now with Linux 5.14 recently minted, he’s been self-reflecting given the “depressingly large” changes at hand. He shared today that he’s thinking of ending his -ck / MuQSS effort so for now at least no updates are planned past the existing Linux 5.12 patches.

Con wrote on his blog, “I’m once again left wondering if I should be bothering with maintaining this patch-set, as I’ve mentioned before on this blog. The size of my user-base seems to be diminishing with time, and I’m getting further and further out of touch with what’s happening in the linux kernel space at all, with countless other things to preoccupy me in my spare time.

He further added, “There is always the possibility that mainline linux kernel will be so bad that I’ll be forced to create a new kernel of my own out of disgust, which is how I got here in the first place, but that looks very unlikely. Many of you would have anticipated this coming after my last motivation blog-post, but unless I can find the motivation to work on it again, or something comes up that gives me a meaningful reason to work on it, I will have to sadly declare 5.12-ck the last of the MuQSS and -ck patches.


Scheduler Changes For Linux 5.15 – Still No Sign Of Any Intel Thread Director Optimizations


LINUX KERNEL --

Ingo Molnar began sending in his pull requests bright and early as usual for the just-opened Linux 5.15 merge window. With the scheduler changes for this next kernel version there are some improvements worth mentioning but also worth mentioning is what hasn’t found its way to the kernel yet: any software optimizations around Intel Thread Director for upcoming Alder Lake processors.

The new scheduler material for Linux 5.15 includes changes for dealing with asymmetric scheduling affinity. This asymmetric scheduling affinity is initially focused around handling of 32-bit tasks on AArch64 systems where some SoCs are having AArch64-only cores not capable of 32-bit (AArch32) execution. The scheduling changes allow defining their own CPU possible mask for tasks to ensure the scheduler will place a given task on a CPU that supports it. Again, initially all focused on the Arm front with legacy 32-bit tasks for some SoCs having 64-bit-only cores.

The scheduler changes for Linux 5.15 also add cgroup SCHED_IDLE support, deadline scheduler improvements, enhanced CPU node-distance determination, and various fixes. The full list of scheduler patches for the Linux 5.15 merge window can be found via this PR.

Notably what isn’t part of this pull request nor have I seen it elsewhere on the kernel mailing list or any prominent staging public Git repositories is any Linux support/optimizations around Intel Thread Director. With upcoming Alder Lake processors there is Thread Director as the new Intel hardware-based functionality for trying to determine the best placement of a given task between its mix of E energy efficiency and P performance cores.

Thread Director is hardware-based for trying to determine the most appropriate task placement among Alder Lake and future Intel hybrid processor designs, but there is a software element at play too. Intel made clear back during Architecture Day that Windows 11 will carry optimizations for Thread Director but wasn’t too clear on the specifics. Intel has also been mum on any Linux software support/optimizations around Thread Director. Well, with no patches queued up for Linux 5.15 that in turn will be out as stable this autumn and with the first Alder Lake processors due out later this year, it doesn’t look like Intel will have any launch-day Linux optimizations in place.

The Linux kernel has long been catering to Arm’s big.LITTLE designs and supporting features around energy aware scheduling and other software improvements on that front, including this work in 5.15 around proper scheduling of tasks if certain cores have reduced capabilities, but we haven’t seen anything on the Intel front in the scheduler or power management areas. P-State has prepared for Alder Lake / hybrid CPU designs but again no kernel activity around the Thread Director front even in early patch form on the LKML.

Thread Director should work fine without any OS engagement given that Microsoft Windows 10 should work fine with Alder Lake without any apparent kernel changes, but in any case we’ll see what comes up in the weeks/months ahead and how the Alder Lake Linux performance is out-of-the-box later this year.


Linux 5.14 Released With New Hardware Support, Core Scheduling, MEMFD_SECRET


LINUX KERNEL --

As expected Linus Torvalds promoted Linux 5.14 to stable in providing the latest features, hardware support, and other improvements ahead of the autumn 2021 Linux distribution releases.

See the Linux 5.14 feature list for a comprehensive list of the changes in this new kernel version. Some of the Linux 5.14 highlights include core scheduling support, secret memory areas support with MEMFD_SECRET, continued enablement around Intel Alder Lake, Yellow Carp and Beige Goby AMD graphics support, AMD SmartShift laptop support, Raspberry Pi 400 support, and more. Linux 5.14 has the usual mix of new hardware support, improving existing features, and adding in other new kernel innovations.

This Linux 5.14 kernel release comes just days after the 30th anniversary of Torvalds announcing the Linux kernel. Linus Torvalds wrote in today’s Linux 5.14 announcement, “So I realize you must all still be busy with all the galas and fancy balls and all the other 30th anniversary events, but at some point you must be getting tired of the constant glitz, the fireworks, and the champagne. That ball gown or tailcoat isn’t the most comfortable thing, either. The celebrations will go on for a few more weeks yet, but you all may just need a breather from them. And when that happens, I have just the thing for you – a new kernel release to test and enjoy. Because 5.14 is out there, just waiting for you to kick the tires and remind yourself what all the festivities are about.

Now it’s on to the Linux 5.15 merge window with a lot of exciting changes planned.