🔈 Kernels, pipewire, and xruns in Linux
TL; DR Uninstalling the
-rt kernel fixed the latency problems in the end, and
rtkit is little understood and hard to configure (on Arch-based things…). Now for the long version…
I have an older laptop now dedicated to music things including realtime audio.
🎹 Excellent apps like Jamulus and Sonobus are usable with decent wired internet (a whole other post perhaps), but in doing so you eventually want to shave off latency (and unreliability) on your audio system too. Using Pulseaudio is never going to get there, as any user of Jack knows. But what if…
In praise of Pipewire
Pipewire is the not-so-new wunderkind of Linux audio. It truly is a great project with a huge amount of momentum, and in my experience so far, fulfilling the dream marriage between Pulseaudio, Jack, and much more. In practice its provision of both PulseAudio and Jack drop-in replacements is game-changing. This is before we even get to its multimedia features.
Also interesting to see all the Rust code / bindings and how that part of audio community will evolve.
…and also Manjaro
As a long-term Ubuntu user and fan, it was hard to part, but the time had come and the draw of Arch-based distros
Given that time is short, and in some way
Manjaro : Arch ≅ Ubuntu : Debian then it seems like the right choice. Installation was great, other than some pretty scary LUKS moments (see various helpers).
Naturally, next step was to install lots of packages (including pipewire and audio utils) with the excellent
pacman. And using
manjaro-settings, a new
-rt kernel was also easy to install too. So maybe that’s enough?
Why might we need low latency?
In short, realtime audio (e.g. playing an instrument) strives to reduce the delay. Humans are quite good at noticing this, and it’s cumulative: audio + network + processing…
OK so it’s not just low latency
With decent internet and gigabit networking, we can reduce the jitter massively compared to Wi-Fi especially. This is pretty important as humans are pretty good at adapting, including adapting to a fixed latency1
The thing that really hurts is variable latency, especially sudden weird pauses. So garbage collection, rebuffering, etc are to be avoided.
So why all the
Wait, what’s an
ℹ️ That’d be short for (buffer) underrun, or perhaps overrun I guess. All those pops and crackles that annoy you.
Essentially other processes can get in the way of delivering all that audio at say 48KHz, (or perhaps way more). This will happen a lot more if these other processes are a higher priority than the audio-delivering process. And this is where the kernel starts to get involved…
Now pipewire uses rtkit to promote its threads to realtime, for better / more predictable latency.
However, I started seeing lots of these errors
rtkit-daemon[...]: Failed to make ourselves RT: Operation not permitted
Which seem to be about rtkit not having sufficient Nice / RTIO privileges to make itself high priority, which it needs in order to manage and monitor other processes…
There is a wider point about using rtkit with realtime audio applications even when it is working as intended..
systemd on Manjaro
So maybe we need to change some startup parameters and config for
rtkit-daemon then? With
systemd I guess?
It’s not quite the same as elsewhere – on Ubuntu the standard system-level config lives at
But there’s a whole bunch of stuff in
/etc/systemd/system/, and setting the environment variables reliably and correctly isn’t easy either.
The thing is, any amount of fiddling with the
NICE configuration for the
rtkit unit didn’t have the desired effect, though I learned much more about
limits.conf along the way.
From The 5.10 Kernel docs:
Enabling CONFIG_RT_GROUP_SCHED lets you explicitly allocate real CPU bandwidth to task groups.
Oh, err right. but also – how did we get there? The clue that this might be relevant at all came up in this old Arch bug, detailing how we don’t want CONFIG_RT_GROUP_SCHED and in fact how to find what your kernel has it set to:
zgrep CONFIG_RT_GROUP_SCHED /proc/config.gz
This returned that it was… turned on in the current system / kernel. Kernels, you say?
Jeeeez there are a lot of these things now that sound like the kind of things you want.
And it’s not always clear which one you want, esp in 2022, given the overlap and the complexity of the space. It’s definitely complicated at the application level, too.
Back in the game
Sometimes the simple option - let’s just try the stock kernel 2 (thanks GRUB)… and whaddyaknow, the rtkit problems are gone 🎉
As per the screenshot – network ping is excellent, and latency is back to being good. So, err, the only thing to blame is my own playing now…
So with many, many hours / days wasted, what became clear to me is: no matter how tempting, you probably don’t want the
-rt kernel at all. 😬