rowyn

Linux is Too Big To Survive,
Or How the Bazar built a Cathedral.

Recently I had that regular occurrence for any sysadmin of needing to rebuild and migrate a disk array. And during that I decided to see if vanilla[1] Linux's LVM and BTRFS raid solutions were fit for use. I eventually found they are not, and had to rebuild the array once again. A combination of poor performance, no RAID5/6 consistency guarantee, and over complex management needs and ability to eat all the data in the entire array if you reboot during a disk migration.

And yet, somehow BTRFS and LVM are used by Amazon and Synology in production and consumer systems. So how does that work? Well, it's because they use their own customised versions of Linux with special sauce added that fix up the problems extant in the vanilla Linux kernel. If you've ever pulled the disks from a Synology NAS, you'll find that the on disk format is almost but not quite identical to that used in vanilla Linux, and different enough to cause problems recovering data from them. And importantly, there's no real interest from Synology and Amazon to get their special sauce into the vanilla source. It's a nice competitive advantage to have their proprietary system that works well and no one else has. It's very hard to work out from what Synology publish, how to apply their changes back to the to Linux. And Amazon don't have to distribute their internal changes at all because they keep it entirely in-house and don't let others use it. And other companies do the same with so many other parts of Linux, and some of them[2] do so with inscrutable compiled binary[3] distributions that go against the whole idea of open source development.

So how did we get here? Two reasons. The first is that in the old days of open source operating systems, hardware manufacturers had no interest in developing device specific system software (device drivers) for niche hobbyist operating systems. The expectation was that at most you might get the technical specifications of the devices, or you might have to reverse engineer the device to work out how the operating system needs to talk to it. This lead to Device Drivers being developed in and bundled with the Open Source Operating System Kernels, or developed by what ever niche interest group needed them. This was, for a long time, how the Linux Kernel operated having inherited this from previous open source operating system projects. There would be the "in-tree" developed hardware drivers managed by the Linux Kernel Mailing List (LKML), then there would be other "out-of-tree" developed hardware drivers managed by the likes of Linux-TV and V4L groups that had a niche interest in broadcast media and CCTV hardware. And the occasional hardware producer that wanted to have a free embedded OS on their device.

And that last one is where things went a bit screwy. A company called Tivo created one of the first Set Top Boxes that could record, replay, pause and rewind LiveTV. And it ran Linux inside the box. But they only published the bare minimum on how the OS ran, and had controls to prevent the device's operating system being directly accessed or altered.

For many LKML developers this was viewed as an Existential Threat. And one that coincided with the introduction of secure-boot[4] methods, intended to lock down things such as public terminals and office desktops from third parties trying to interfere with the OS. The LKML named this "Tivoization" and declared it something to be combated. First they started to argue that these were violations of the 'GPL2'[5] document used as a license to distribute Linux. It should be noted that these arguments came from software engineers, not lawyers. In addition they encouraged out-of-kernel developers to merge their organisations into the LKML, eventually resulting in official policy that the LKML would not take into account any issues their changes might make to out-of-kernel drivers.

This led to the current situation, Linux has a policy that it does everything possible to avoid breaking the applications that run on Linux. It has no such policy for out-of-tree developed drivers, and those can not only be broken by changes between major versions of Linux, but can sometimes also be broken by changes to the 'Long Term Support' versions. But more importantly, it did nothing to stop "Tivoization". There are now plenty of small embedded devices running Linux that include binary blobs and boot time intrusion prevention. Your TV might be one of them, your home router almost certainly is.

The result of this is that now the vast bulk of the LKML distributed Linux Kernel is Device Drivers. And all of this code is in variable states of maintenance and quality. Occasionally they will prune some of the drivers after deciding that no one running a current version of Linux needs them. Sometimes they just sit there for ever. Sometimes they unintentionally became support structures for other drivers and can't be removed without massive restructuring. In the world of computer systems this is called "technical debt", old decisions that cause problems for future development. And if you want to get your driver into the vanilla kernel and be a sanctioned in-tree kernel driver, you need to abide by the design and conduct choices and social structure of the LKML. Suffice to say that discussion of what that means is out of scope for this article.

There are now very few major out-of-tree but still open source Kernel Drivers. Returning to the original paragraphs of this article, one of them is OpenZFS. OpenZFS is a high performance file system intended for use with disk arrays and gives a highly functional and robust RAID and disk recovery implementation. It also inherited a software licence from it's original project that while still fully open source, is considered legally incompatible with the GPL2 by the LKML developers, so it can never become an in-tree driver. This is despite the licence actually being a stronger open source protection, as it prevents any of the code submitted being patented by the submitter. OpenZFS is a substantially better disk array file system than BTRFS and LVM on Linux. And it has major industrial backing, because the ZFS file system is deployed heavily throughout enterprise systems. But if you want to use it on Linux, it means you have to use OpenZFS's out-of-tree drivers. Which means you can only use the specific versions of the Linux Kernel known to safely work with OpenZFS. Try to replace your Linux Kernel with a new one that has an incompatibility, and you lose access to the disk array. The consequences are the same for smaller niche hardware that doesn't have the backing to get through the LKML gauntlet to become an in-tree driver.

The proponents and supporters of the GPL2 supported it as being an alternative to giant corporate close-source monolithic development. That instead of a Single Software Cathedral, you could have a bazar marketplace of smaller developers each contributing to individual, joint or distributed projects. But the irony of the LKML interpretation of GPL2 and the organisation practices of the LKML has been to create a new Cathedral. One that is accruing vast amounts of technical debt.

So what is the future of Linux with all this technical debt hanging over it?

First possibility is that it gets hard forked. Which means someone steps in, takes a copy all the public code (called forking the source tree), creates a new organisation, and starts developing a new Linux with different policies. And this could happen multiple times, with all different interested groups making their own Linux. And the thing about this is that it's the one that is already happening. Android is a hard fork of Linux sponsored and organised by the Alphabet corporation to run on phones and TVs. The majority of corporate Linux distributes manage a soft fork of the Linux kernel with patches and maintenance reviews of their own. Ubuntu and Red Hat ship their own versions of the Linux Kernel Source Code. It's quite possible that eventually this turns into a hard fork. If this progresses, then eventually the LKML lose relevance as they become more and more detached from where Linux is used.

Second possibility is that the LKML slims down. That they abandon the policy against supporting out-of-kernel development, and spin off the device drivers to independent projects. In some cases this would be spinning them off when they had been independent projects in the first place. Development by 'Special Interest Groups' is actually the way almost all major standard hardware systems get developed. Combinations of corporate, government and academic interests combine to work together on a project researching and developing a new technology. The LKML actually has very little sway on hardware development, because it simply isn't organised to recognise that special interest groups exist outside the LKML. Currently hardware is near universally developed without any regard to the LKML[6]. By divesting of driver development to special interest groups, open source operating development would actually get a stronger voice on hardware development. In a sense, this too also happens in that some parts of driver development are allowed their own soft-fork that is managed under the LKML organisation, with changes back-ported. One example is the DRM-Next source tree, which handles Graphics Drivers where a powerful enough group of special interest exists. Out of kernel development of drivers that can be packaged alongside a kernel used to work in the past, it can work again now. It just needs the LKML to decentralise and spin off, and start to get a handle on dismantling the technical debt.

Or maybe something in the middle happens. And we end up with semi-soft forks of the Linux Kernel, organising Special Interest Groups. But in that situation the LKML will struggle on under the technical debt of trying to include as many in-tree drivers as possible while not supporting out-of-tree special interest developments.

What I hope is that Linux doesn't collapse under completely unmanageable technical debt. And it fracturing into corporate owned distributions of their secret special sauce versions.

--
[1] A Linux kernel compiled from the publicly distributed source code release from the Linux Kernel Mailing List operated kernel.org repository. For reasons discussed, this is not the same as the Kernel that is actually used by anything other than hobbyist distributions.
[2] The notable example being Nvidia.
[3] For non-technical readers, a compiled binary is the raw machine level instructions. They are generated from the computer program's 'source code' designs that developers create, and binaries can be hard if not impossible to deconstruct back to their original design work. Think of it like trying to identify the ingredients in a can of soup without the label. To even start, you need a special tool to open the can.
[4] Opposition to Secure Boot was based on some faulty assumptions about the intent behind it, and ignoring the clear needs when you have devices that are publicly physically accessible but need to handle confidential data such s payment processing. Eventually, and with a push from third party vendors, Secure Boot was embraced by the LKML.
[5] GPL2 can be read so as to imply that you can not distribute incompatibly licensed software that 'links' to a binary API because that makes it 'part' of the licensed software. Even where the license is also an open source licence. This is disputed, and it is in my view not something the licence can legally establish as a binding term. However, LKML developers have taken to marking certain API calls to be accessed by GPL2 software only. This is of course only obeyed by people trying to respect the LKML developers opinion on this.
[6] There has been a fair amount of antagonism between Hardware standards development and the LKML, due to a tendency to make public statements decrying flaws in hardware designs after those hardware designs had been set in silicon for years and are now hard to change. At one time it was a common refrain on the LKML that users should 'fix their hardware'.