🔈 Kernels, pipewire, and xruns in Linux

Posted June 2022

TL; DR Uninstalling the -rt kernel fixed the latency problems in the end, and rtkit is little understood and hard to configure (on Arch-based things…). Now for the long version…

Playing music

I have an older laptop now dedicated to music things including realtime audio.

🎹 Excellent apps like Jamulus and Sonobus are usable with decent wired internet (a whole other post perhaps), but in doing so you eventually want to shave off latency (and unreliability) on your audio system too. Using Pulseaudio is never going to get there, as any user of Jack knows. But what if…

In praise of Pipewire

Pipewire logo

Pipewire is the not-so-new wunderkind of Linux audio. It truly is a great project with a huge amount of momentum, and in my experience so far, fulfilling the dream marriage between Pulseaudio, Jack, and much more. In practice its provision of both PulseAudio and Jack drop-in replacements is game-changing. This is before we even get to its multimedia features.

Also interesting to see all the Rust code / bindings and how that part of audio community will evolve.

…and also Manjaro

As a long-term Ubuntu user and fan, it was hard to part, but the time had come and the draw of Arch-based distros

Given that time is short, and in some way Manjaro : Arch ≅ Ubuntu : Debian then it seems like the right choice. Installation was great, other than some pretty scary LUKS moments (see various helpers).

Naturally, next step was to install lots of packages (including pipewire and audio utils) with the excellent pacman. And using manjaro-settings, a new -rt kernel was also easy to install too. So maybe that’s enough?

Why might we need low latency?

In short, realtime audio (e.g. playing an instrument) strives to reduce the delay. Humans are quite good at noticing this, and it’s cumulative: audio + network + processing…

OK so it’s not just low latency

With decent internet and gigabit networking, we can reduce the jitter massively compared to Wi-Fi especially. This is pretty important as humans are pretty good at adapting, including adapting to a fixed latency1

The thing that really hurts is variable latency, especially sudden weird pauses. So garbage collection, rebuffering, etc are to be avoided.

So why all the xruns?

Wait, what’s an xrun?

ℹ️ That’d be short for (buffer) underrun, or perhaps overrun I guess. All those pops and crackles that annoy you.

Essentially other processes can get in the way of delivering all that audio at say 48KHz, (or perhaps way more). This will happen a lot more if these other processes are a higher priority than the audio-delivering process. And this is where the kernel starts to get involved…

rtkit-daemon

Now pipewire uses rtkit to promote its threads to realtime, for better / more predictable latency.

However, I started seeing lots of these errors

rtkit-daemon[...]: Failed to make ourselves RT: Operation not permitted

Which seem to be about rtkit not having sufficient Nice / RTIO privileges to make itself high priority, which it needs in order to manage and monitor other processes…

There is a wider point about using rtkit with realtime audio applications even when it is working as intended..

systemd on Manjaro

So maybe we need to change some startup parameters and config for rtkit-daemon then? With systemd I guess?

It’s not quite the same as elsewhere – on Ubuntu the standard system-level config lives at /usr/lib/systemd/system/rtkit-daemon.service

But there’s a whole bunch of stuff in /etc/systemd/system/, and setting the environment variables reliably and correctly isn’t easy either.

The thing is, any amount of fiddling with the RTPRIO and NICE configuration for the rtkit unit didn’t have the desired effect, though I learned much more about limits.conf along the way.

What is CONFIG_RT_GROUP_SCHED 🤔?

From The 5.10 Kernel docs:

Enabling CONFIG_RT_GROUP_SCHED lets you explicitly allocate real CPU bandwidth to task groups.

Oh, err right. but also – how did we get there? The clue that this might be relevant at all came up in this old Arch bug, detailing how we don’t want CONFIG_RT_GROUP_SCHED and in fact how to find what your kernel has it set to:

zgrep CONFIG_RT_GROUP_SCHED /proc/config.gz

This returned that it was… turned on in the current system / kernel. Kernels, you say?

Linux Kernels

Jeeeez there are a lot of these things now that sound like the kind of things you want.

  • -rt
  • -realtime
  • -preempt
  • -lowlatency

And it’s not always clear which one you want, esp in 2022, given the overlap and the complexity of the space. It’s definitely complicated at the application level, too.

Back in the game

jamulus audio screenshot

Sometimes the simple option - let’s just try the stock kernel 2 (thanks GRUB)… and whaddyaknow, the rtkit problems are gone 🎉

As per the screenshot – network ping is excellent, and latency is back to being good. So, err, the only thing to blame is my own playing now…

So with many, many hours / days wasted, what became clear to me is: no matter how tempting, you probably don’t want the -rt kernel at all. 😬


  1. remember the rule that 1 foot of distance ≅ 1ms of delay, so even acoustically a stage itself could add 20ms.↩︎

  2. yes, in rage at wasted effort I then purged the -rt kernel from my system completely afterwards 😠↩︎