Transcription
Hello. In this video, I would like to take a look at how sound on Linux works and how all the different pieces that you might have heard of, like ALSA or OSS or PulseAudio and JACK, come together to form the complete Linux audio ecosystem.
First, let's start with the basic hardware. The whole point of the sound system is to be able to interact with physical audio inputs and outputs. Outputs from the computer will go into usually speakers or headphones, and inputs will be from things like this microphone or from other line inputs, like other computer audio devices.
The first thing we need to interact between the software and the hardware audio devices is an actual sound card. This is a card that's going to have all the required digital-to-analog conversion to be able to take the digital sound from the computer and change it to analog sound for headphones or speakers, as well as analog-to-digital conversion, so that you can take the analog signal coming from this microphone and record and record it into a digital audio format. Generally, that digital audio format happens to be PCM, although alternatives such as DSD do exist. Chris Montgomery from zif.org has an excellent explanation of how PCM works, and I will link it down below.
Sound cards also tend to have a bit of additional hardware, you know, stuff like headphone amplifiers so that we can get a decent volume, input and pre-amplifiers so that you can give phantom power for a microphone or be able to boost the volume to where you need it to be. But in practice, it's basically a way to convert analog audio to and from digital.
Next, in the kernel, we need a driver to be able to actually interact with the sound card. It's great that we have this device that can send and receive analog audio, but it's not, it's not much use if we cannot interact with the digital part of it. So the whole point of a device driver in the Linux kernel is to actually be able to interact with the sound card, so we can send bytes to it and read bytes from it to be able to like read and write to audio devices.
In the early days of Linux, this was commonly done with a driver called OSS, which stands for Open Sound System. Unlike the actual name, OSS actually was proprietary for some time. Um, started out was open and became proprietary, so on and so forth. It also had some limitations.
The replacement for OSS is what's in use today. ALSA stands for Advanced Linux Sound Architecture, and it's basically what supports everything from USB sound cards to PCI sound cards to onboard audio on your laptop and desktop. ALSA provides device drivers necessary to be able to read and write from a sound card. It doesn't provide much more than that. It's just a way to interact with a sound card.
Many applications support working with ALSA directly. If I pop open a video in VLC media player, it can actually output audio straight to an ALSA device. Similarly, Audacity can directly record from an ALSA device. In practice, this isn't usually what you do, though, because some basic sound cards don't actually support multiplexing. So while it's great that we can read and write from it, if a program is using ALSA, it'll just take control of the entire sound card, so only one program will be able to play or record audio at a time, which in general isn't what you want. You want to be able to have music playing while you play a video, for example, or you want to be able to have two separate, like programs running that both want to output or record sound. You know, it would be silly if I had OBS running and also had Teams, except that only one device could use a microphone at a time.
But this used to be a real limitation. In fact, many sound cards advertised having support for hardware mixing. Hardware mixing is when you would literally, each program would quite literally send its audio stream to the sound card, and the sound card would have the job of taking these and mixing it into a single output that can be sent to speakers or headphones.
Now, ALSA is great, it's robust, but that limitation of not being able to multiplex sound cards properly has led to replacements. Well, not replacements, ALSA still remains a driver, but we have abstracted a layer above the device driver these days that happens to be PulseAudio. And more recently, PipeWire is replacing PulseAudio, but we also have JACK, which was for pro audio systems.
Now, both of these are what we call sound servers because the server is going to interact with the sound card, and then the programs are the clients. So each of these client programs can output their sound to the PulseAudio server, and then PulseAudio will take, combine these, do the mixing, and output it to the graphics card.
Now, this has many benefits, as you might imagine, as well as a couple of drawbacks. The benefits are that for no matter what sound card, even one that only supports one end, one digital input, I can have 10 programs open that are all playing different songs or different videos, and PulseAudio will take all of those, resample them, and mix them, and then send them to the sound card.
If you're not sure what resampling is, that's a process where, that's how many samples per second we have. So, for example, audio CDs are traditionally 44.1 kilohertz, so we have 44.1 samples per second. However, digital videos are usually 48 kilohertz, and some audio file recordings might be 96 kilohertz. So what happens by default, if your sound card didn't support resampling, you only, you want the sound card to only to run at a fixed output, say 48 kilohertz. On most modern sound cards, there's no way for the sound card to otherwise take in both a 44.1 and a 48 kilohertz signal without resampling one of them to match the other. And these days, you'll often just take one, anything, and resample it to 48, just because it tends to be the most common we see everywhere.
Now, fear not, the resampling process, as much as some people claim, distorts the audio signal. It's actually a relatively clean process because you can't hear anything about 20 kilohertz anyway. By default, PulseAudio ships the number of resamplers. The Speex ones are used by default for a good balance of CPU power to audio quality, and realistically, if no one ever told you, you probably wouldn't have even noticed that it was being resampled.
Now, the other good benefit of PulseAudio is we can do the reverse for microphone inputs. The sound card can record the input, send it to PulseAudio, and PulseAudio can distribute this to any program requesting it. Now, this is great because this means that my programs, I can then record the same microphone, say, on both Teams and Skype, or both, like Zoom and OBS, and this way, I don't have to have two physical microphones plugged in.
You might have actually noticed this multiplexing in that digital video is still not multiplexed on most computers. This means that if I have a camera, and this camera is being used by guvcview so I can monitor it, guess what? I cannot use the same camera on Zoom. I will need two cameras, usually. And that's just because while CPUs have gotten fast enough to be able to easily multiplex audio, multiplexing at the camera is still a lot of work. PipeWire, which is going to replace PulseAudio, hopefully as soon as possible, does support using GStreamer to multiplex video, but the CPU usage speaks for itself. It's not trivial.
PulseAudio has a few more benefits. For example, if I have two sound cards, most comp, people that have video cards might have noticed that their HDMI output is technically also a sound output. So if I had a TV connected, I could send sound to my TV. Um, so in that case, you might have two different audio devices, and PulseAudio easily provides an API for each program to just send their sound to PulseAudio, and then we can pick which sound card each one will go to. So I can have music on my headphones, but a movie on the TV, so on and so forth. It also lets you individually adjust the volume of each program, which is pretty neat. So you can have the music at a lower volume than the video, for example. Hardware mixing didn't usually support that.
If you're on a Linux computer and you're playing audio, chances are it's using PulseAudio. Now, a competitor to PulseAudio in the pro audio space was JACK. JACK was designed as a virtual, essentially patch panel, and it's way more flexible than PulseAudio is, albeit slightly harder to use. So some drawbacks of PulseAudio were that it's hard to have a program, say, output to two devices at once, or connect any input, input to any output. PulseAudio is a very traditional model of, here are my inputs, here are my outputs. For example, if I want to split my sound card into, I need to make a virtual, two virtual sync devices, and those virtual sync devices will map to a single real output device, that will be the sound card. So, and so forth. It has a lot of these, um, steps.
There's also a second major disadvantage to it, which is the latency. PulseAudio usually has latencies of 40 to 80 milliseconds, and if you're trying to run pro audio tools, that's probably not ideal.
JACK stands for JACK Audio Connection Kit, obviously recursive acronym, because VA Linux programs. So JACK is written in C. JACK 2 is the C++ replacement. They are not necessarily compatible, but they kind of do the same thing. The way JACK worked is it's very similar to PulseAudio in that it's a sound server, so all programs will read and write their sound to and from JACK, and then JACK will actually do the interacting with sound cards. Now, unlike PulseAudio, JACK also supports UART as well as MIDI. MIDI being the important one because it lets you know, a lot of music pro audio tools work, happen to work with MIDI, whether that be MIDI keyboards or like different lighting controllers you're controlling. MIDI is pretty standard musical instrument data, whatever.
So JACK is way more flexible than PulseAudio. I will pop open a JACK graph here so you can see what I mean. JACK is a simple way to connect the output of any program to the input of any other program. As they describe it, it is pretty much like a physical patch panel. If you work at the physical patch menu, you can take outputs from any of these instruments or devices, pop them open to whatever hardware filters you want, or, um, the analog mode of VSTs, take their output, and then you can plug them back in, so on and so forth. You can do any of this mapping where I can start daisy chaining things. JACK kind of lets you do that where I can have the microphone go into some program, and then the output of that can go into some other program. I can split my microphone into multiple programs. I can take all of those inputs and combine them. I can make, downmix them into stereo, so on and so forth.
Some of the other benefits are, it happens to run at a much lower latency than PulseAudio. JACK servers have much shorter delays. They also have the advantage of being able to directly interact more with hardware. So, for example, higher sampling rates in PulseAudio tend to be buggy. They work a lot better in JACK because it was designed for pro audio by pro audio people.
So if it's so much better than PulseAudio, why isn't it used everywhere? Well, it kind of comes down to compatibility. PulseAudio is just easier to work with, it's simpler, and it kind of just works. And for most people, the latency and stuff isn't really not an issue. Realistically, some of the other features offered by JACK, like people don't use MIDI, people are don't need to time synchronize all their programs. It is cool though, because if you're a pro audio person, you can have all of these different recording tools and have them all synchronized on the same clock, whether that be their sound card clock or whatever, and have them all record, pause recording to any number of tracks. For example, Ardour, the DAW that I generally use for recording audio, tends to work a lot better with JACK because you can just capture, if I have eight different microphone inputs, I can just capture all eight of them. And the benefit over using it instead of ALSA is that I can still run whatever else I need to run.
Now, unfortunately, PulseAudio and JACK often aren't compatible with each other, just due to how different the different sound servers are. If you want to run both, generally what you have to do is you have to output PulseAudio as an input and JACK, and then go from the output of JACK to your sound card. This is not optimal for obvious reasons. If I have a program that's playing the PulseAudio, it's gonna have to go program to PulseAudio, from PulseAudio to JACK, and then whatever happens, happen in JACK, and then that's going to go to my sound card. Doesn't really help the latency cause, does it?
So PipeWire is going to hopefully be the ultimate replacement for both JACK 2 as well as PulseAudio. It takes the ease of use and simplicity of PulseAudio, as well as the benefits and advanced feature set of JACK, and kind of combines them into some into a pretty good package. I've been using PulseAudio on my computer for the past four months, and it's been working pretty well, with the exception of some Bluetooth devices being a bit buggy. I'll touch on Bluetooth at the end of this video because it kind of interacts differently. But PipeWire has been a great replacement.
PipeWire offers a drop-in replacement for PulseAudio, which is the important bit. Um, so any program that used to work with PulseAudio will work with PipeWire. That's the great news. I can have Firefox, VLC, MPV, whatever. Every program will kind of just, instead of just connecting to the PulseAudio sound server, it'll connect to the PipeWire sound server, and they have the basically the same PulseAudio API. So no matter which sound server you happen to be using, both will work in the same way. The benefit of PipeWire, though, is how it tends to handle inputs. Instead of PulseAudio's traditional model, PipeWire uses a model that's way more reminiscent of JACK, to the point where it also is a drop-in replacement for JACK programs.
So finally, for once, in the Linux ecosystem, you can run both JACK and PulseAudio programs simultaneously, and they can finally interact with each other properly. This is great news because it brings some of the benefits of JACK to PulseAudio. So this means that now I can use the JACK patch panel approach with the with the JACK graphs, for example, in mini managers, to direct my PulseAudio application. For example, I can finally have the outputs of my sound card split the way I want to, or route any program to any program the way I want to. For example, it makes recording with FFmpeg much easier because I can just easily pick exactly which programs I want to input or output where. It provides that excellent flexibility that JACK used to, with the great compatibility of PulseAudio.
Now, by default, you need to use `pw-jack` when you're launching a JACK program to use PipeWire instead of the default JACK server. But there is a drop-in replacement package on many distributions that will let you directly just run JACK programs using PipeWire server. It is slightly higher latency, and I have noticed a bunch of resampling issues. For example, if you've note, if you have some JACK and PulseAudio application playing together, you'll notice that one of them, because there are issues with the resampling approach being used, I, it's very hard to record that particular, um, artifacting, so I don't think I'll be able to include an example, but it definitely exists. There's definitely some artifacting when going between to and from PulseAudio and JACK applications. Generally, it has to do with 44 versus 48 kilohertz sampling rate, as well as audio buffers not being long enough, because PulseAudio applications want the bigger buffer, but JACK applications can run off a shorter buffer. There's some issues. But all in all, it's a pretty good drop-in replacement, and it kind of takes a spot of both applications. Some stuff, obviously, your config from your PulseAudio server has to be changed, but honestly, for the added flexibility, I think it's a very good catch.
And PipeWire also has a lot of future potential. Some applications, for example, I recently made a video on how to loopback a microphone input, are so much easier on JACK because you can directly connect the input device straight to your output device, and you don't have to go through all of these other fancy hoops that you had to with modules being loaded, so on and so forth. Um, and the future potential of PipeWire is even more promising. So right now, it's an experimental, but hopefully it's going to become stable by the end of this year, or by the time I upload this video. And that is finally, we're going to have the ability to multiplex webcams in the same way that we can multiplex audio.
So as I alluded to earlier, just like how we can have multiple programs playing audio at the same time, ideally, what this is going to allow is for multiple programs to use the webcam at the same time. So I can, as I said, I can have multiple programs use a mic, why not a camera? Well, PipeWire is implementing this through GStreamer, so any program that's going to support GStreamer will hopefully be able to take in your camera feeds and have it multiplexed. Now, as I said, this kind of is still, it's still experimental, it's still buggy, it doesn't work in all programs, and it does have a high CPU cost, but it is definitely the way to move the future moving forward. Um, now that we have the computational power to do digital video as easily as we can do digital audio. So hopefully this is going to allow for us to be able to reuse our camera in multiple applications at once.
All right, before I end this video, I want to do a quick note on Bluetooth because Bluetooth is a mess. Bluetooth is a mess in the same way that that wire was a mess. If you have had to deal with FireWire audio interfaces on Linux, you know what I'm talking about. So modern sound cards, PCI, PCI Express sound cards, are all supported by ALSA. Unfortunately, FireWire all cards often weren't. So this meant that if you had professional FireWire audio interfaces, you would have to use separate drivers. The FireWire OHCI stack comes to mind. So you'd have to have those to actually interact with the FireWire device, and then you would have to load JACK or whatever audio server you're using on top of that, and often it had to be JACK in those cases. Obviously, this is, uh, kind of not ideal. And thankfully, FireWire has kind of gone, well, maybe not thankfully, if I, it was a neat protocol ahead of its time, perhaps. But since FireWire devices aren't really used anymore, that's not really an issue.
What is common these days is Bluetooth, and Bluetooth audio still doesn't always interact properly with ALSA. In many cases, Bluetooth headphones would actually connect directly to the PulseAudio server, not through the ALSA driver. This always has some, this always had some couple quirks because Bluetooth headphones can run in a number of modes: SBC, subband codec being the default, A2DP sync, you have the HFP and HSP and HFP headset profiles, which have horrible sound quality, but are still needed because it's part of the Bluetooth audio standard, as well as on modern headphones, we now have AptX, AptX HD, and AAC audio. So in many cases, these Bluetooth devices have the same problem where, due to the various codecs supported, so on and so forth, um, you know, support kind of varies. Sometimes you need to repair the headset each time. And PipeWire is no better, unfortunately. I have had more buggy experiences with Bluetooth running on PipeWire than I have with PulseAudio. And so it's kind of the one last downside for me. Thankfully, I don't use Bluetooth headphones very often. I tend to like the reliability of just wired devices and not having to recharge them, and to better sound quality as a bonus, I guess.
With PipeWire, there are some issues where, for example, you turn the device on, first time it pairs, but if you turn it off and back on, it won't always connect to the sound server properly. PipeWire will detect it as being there, and it, it'll output audio, but the Bluetooth device doesn't actually receive any audio. Sometimes it works by changing the codecs, sometimes it doesn't. It's a bit of a mess. Hopefully, by the time I upload this video, some of these issues will be resolved. I know there are many bug reports that are going in weekly onto the GitLab repo for PipeWire, and many of these issues are getting fixed. So hopefully this will just be a short-term hurdle.
With that said, thank you so much for watching this very long turbo nerd edition video on Linux audio and how sound servers versus device drivers versus sound cards and how it all kind of works. These videos really aren't meant to be for a mainstream audience. This is kind of more for people like me where you kind of just get curious about something and you do kind of do a bit of reading, or you're kind of just curious about how it works. It kind of happened to me when I was stopping and thinking about like how all of this, uh, how audio works at all on Linux.
Thank you for watching. If you liked this, if you learned something, consider liking the video. Consider subscribing to my channel. If there's anything I missed, feel free to leave it in the comments. I'll try to make an errata or correction in the description and in a pinned comment somewhere. But thanks once again for watching. Have a great day. See you next time.
[Music]