After they repurposed the "debug" kernel flag and made it literally unusable with systemd, Linus banned one of their core devs (Kay Sievers) from contributing to the Linux kernel, and I quote:
"I'm not willing to merge something where the maintainer is known
to not care about bugs and regressions and then forces people in other
projects to fix their project."
It's worth noting that this is a case of Linus flying off the handle based on incomplete information. It was latter revealed that the bug which caused this whole spat on LKML had already been fixed in systemd, Kay Sievers just failed to communicate that in the bug opened by Borislav Petkov.
1. A system hang due an assertion in journald spamming kmsg.
2. The question of whether systemd should be logging to kmsg or doing anything based on the "debug" kernel command line argument.
Linus got pissed because, based on Steven Rostedt's post to LKML and the referenced bug report, it looked like Kay was refusing to fix issue #1, but that bug had already been addressed and a fix committed to systemd. In the aftermath there was also a flamewar over issue #2, which is what Lennart is responding to in that G+ post.
I mean rate limiting stuff that comes from anywhere is needed. not cause of the fact that systemd does it.
I mean if I could write a userspace tool that crashes the kernel.. is somewhat aweful.
The place have become something of a second home for a lot of Linux related devs. One potential problem with that is that the poster is free to delete any and all comments made to his posts.
People have been trying to fix the boot process in linux for years. Like most things, everybody screams when something changes, but nobody is willing to put in the work to make something better.
RedHat had enough, and finally jammed systemd down on the Linux ecosystem.
This had two effects:
1) Technical: it exposed a LOT of shortcomings in the architecture of Linux for operating on modern systems. The whole "This belongs in PID 1! No it doesn't!" stems from there not being good, correct, and obvious ways of accomplishing the tasks that need to be done.
2) Political: it pissed off everybody from Linus down who actually thought they had power and control by demonstrating forcibly that their opinions don't really matter.
As for the technical issues, I'm not terribly sympathetic. Somebody needed to fix this, and it was painfully clear that nobody was ever going to get consensus on this. In addition, Linus blocks a LOT of stuff attempting to evolve the kernel in directions that are improvements, but that he does like or doesn't understand. For example, Linus didn't handle the ARM board issues very proactively. He basically ignored things until they got untenable and then finally blew up at the ARM guys and threatened to rip them out of the kernel. People had tried several times to get a modular configuration system put in place, but Linus never wanted to commit it as there "wasn't consensus". True, but once his pants caught on fire, he didn't give a shit about consensus anymore.
As for political issues, I have even less sympathy, as RedHat simply did what Linus has been doing for years and imposed their will on the ecosystem by force. Enlightened dictatorship is a wonderful governmental mode, until you're no longer the dictator. Perhaps if Linus had figured out how to actually build consensus, I would find his words less hypocritical and more deserving of consideration.
The BSD's are no strangers to controversy. The whole existence of OpenBSD and NetBSD is a tribute to that. Even with FreeBSD, the GEOM subsystem changeover caused great friction. The difference is that Poul-Henning Kamp did the work and got a signficant level of consensus from the BSD leadership--but by no means unanimous and it was sometimes acrimonious.
I don't want to argue about systemd vs this or that, but re: (2) you are totally misrepresenting linus's objection to systemd (developers).
the well-publicized incident to which you're referring had systemd introduce something that broke userland, then instead of fixing it in systemd code, proposed a kernel change for it. this is, obviously, bad practice.
it would be one thing if the systemd code exposed a vulnerability or inefficiency in the linux kernel. but fixing your bad code by modifying the kernel is like trying to modify a popular compiler because some bad code you wrote isn't doing what you want it to.
I wasn't referring to any particular systemd thread (my experience has been watching Linux/Linus in the context of ARM), but I think you are referring to the post by semi-extrinsic and this thread: https://lkml.org/lkml/2014/4/2/420
I examined that thread, and the upshot seems to be that the kernel wasn't rate limiting logging so userland could break the kernel. And Linus threw a hissy fit instead of actually examining the situation. The kernel not rate limiting is a fault in the kernel. The fact that a user program exposes it does not mean that the user program is the one in the wrong. And Sievers basically said: "Not rate limiting logging is a kernel problem. We're going to make you fix it." So, basically he had the temerity to both A) pull a political power play and B) be technically correct--and it pissed Linus off ferociously.
Linus, however, is fighting with the 800lb gorilla. That's not a good position to be in. Linus needs the RedHat programmers more than the RedHat programmers need Linus. They'll keep Linus around to avoid the bad PR that would come with a fork, but, if he gets in the way, they'll just cut him out of the loop.
> And Linus threw a hissy fit instead of actually examining the situation.
You really, really, really need to take ten minutes or so and read the whole thread. If you've not the time for that, then the single message linked here provides a decent amount of context: https://news.ycombinator.com/item?id=10484317
I did read the thread before I even posted, thanks.
Linus' first reaction to systemd accidentally flooding the kmesg queue was to make an ad hominem flame directly at another developer when the kernel was responsible for at least half the problem (lack of rate limiting).
Most people would qualify that as "throwing a hissy fit".
> Linus' first reaction to systemd accidentally flooding the kmesg queue was to make an ad hominem flame...
Are you talking about this?
"Key, I'm fcking tired of the fact that you don't fix problems in the code you* write, so that the kernel then has to work around the problems you cause."
If you are, then you have to know that he said that because Sievers has a history of aggressively refusing to own up to (and fix(!)) the bugs he introduces into his code. [0] Contrary to popular understanding, Torvalds doesn't dress down people unless they've demonstrated that they should know better and continue to fail to meet their demonstrated potential.
Most people call that sort of management style "stern" and "meritocratic".
[0] Indeed, it is this behavior that led to Sievers's Linux kernel commit bit being unset. Because of this, Greg-KH had to become the front-man and shepherd for the kdbus project.
I think the way you did. Many things in systemd aren't as good as they should be. However it's working and does his job.
The issue that raised up here is definitly a problem in the kernel. I mean it's not a good practice that other programs could flood yours. However the problem whey they argued that so hard is caused by the fact that both parties trying to fight a political war over the linux ecosystem which is just bad, kay sievers also said it.
They make a project that can only be good if all parties, kernel, init, gui, are playing together and not fighting against each other like childish politicans.
What I don't get is, if systemd is so troublesome, why are so many distros picking it up? I know popularity isn't a perfect signal, but in this case of highly technical users that are distributing OSes it seems valid.
Part of the reason is that systemd has absorbed functionality of a number of additional pieces of software, to the point that they are no longer maintained discretely - udev being the best example.
Another part of the reason is that Red Hat forcibly landed systemd in Fedora and then RHEL7, and RH is an elephant on the scale of Linux development.
So RedHat's choices impact Debian/Ubuntu, Arch, and SUSE so much? Honest question; I don't know the details. Or are you saying that distros rely on other components that are simply not feasible (maintained) anymore, so they have no choice?
Just seems like if it's as bad as so many say, it just doesn't make sense for all these distros to blindly go along. Even the GNOME lockin doesn't seem like it'd explain it.
Debian isn't really a leader. They're more of a passive target platform and their committee has people from various strokes of the Linux community. As such, RH decisions with significant influence definitely would impact them. Ubuntu, in turn, is symbiotic with Debian, though still quite forked from it in most aspects beyond the packaging infrastructure (now with Snappy diverging even further). Nonetheless, Unity needs GNOME and Canonical are still a small player who are perfectly capable of foreseeing future trends. Adopting systemd is the path of least resistance and will help them track Debian's packages better.
Most RPM-based distros (openSUSE included) tend to follow RH's direction, so that's not surprising. Besides, SUSE has always been enthusiastic about most desktop efforts.
Also, distros have blindly went along with other bad ideas before. The most prominent examples were HALd and LSB-style initscripts.
In fact, I'm not sure why you're at all surprised. Large groups of people in real life have collectively made far, far more catastrophic decisions. Why would a bunch of distribution maintainers adopting a piece of software be so shocking?
Systemd is pointing out the truth about lots of "emperor's clothes" in the current Linux ecosystem.
When there were problems about a desktop manager revamp or some crashy audio daemon, you could liquidate certain choices as irrelevant or lazy. Now the entire ecosystem's guts are being rewritten by RedHat for RedHat, and the developer community is simply going along because, in practice, they have no other choice. Nobody else has the appetite, the manpower or the political weight to put together a competing project with a single chance of success.
When chips are down, Red Hat owns Linux more than we like to admit.
I suspect many didn't see it coming because systemd was under the freedesktop.org umbrella rather than a Red Hat fronted project.
Thing is though that while freedesktop.org is presented as being about cooperation and compatibility between desktop environments, a very large portion of what happens there is dictated by Gnome.
And Gnome is yet another project that on paper is independent, but with big RH contributions in terms of programming manpower (its the primary DE of both Fedora and RHEL).
Probably the worst part is that none of this is planned, there is likely no grand conspiracy. Its just that so many of the people involved walk in the same halls, share the same cafeteria tables, and sit in on the same meetings that a internal consensus ends up formed about what is the "right" way to do things.
> Probably the worst part is that none of this is planned, there is likely no grand conspiracy. Its just that so many of the people involved walk in the same halls, share the same cafeteria tables, and sit in on the same meetings that a internal consensus ends up formed about what is the "right" way to do things.
This. The whole Linux community seems to be paranoid. It has to be a grand scheme, a plot to take over Linux, a conspiracy to kill "the UNIX way". Maybe it's just a group of guys solving their problems.
I don't see how anything of this is really Red Hat's "fault". Seriously. They develop a piece of software which solves their problems. They also incorporate projects they already maintain to improve functionality and make developing things easier. It's their projects, they're allowed to do that. The code is still open. It's not like everybody (Debian, Ubuntu, SUSE, etc) didn't have the ability to fork udev.
Yet all these projects chose not to fork the projects incorporated by systemd. Reason (1) might be it solves problems for them as well. There's no need to fork something that perfectly works for you. Reason (2) might be they don't have the resources. But that's a declaration of bankruptcy for the whole "Linux is built by community volunteers" thing. The people advising against "corporate influence" are not able to do anything of value in modern computing anymore (I'm well aware this is over-simplification, please bear with me). Requirements for software have changed. It's not 1970 anymore. And while the "old-school" users and devs are complaining about systemd & co, the "youngsters" are busy changing the ecosystem.
> (2) might be they don't have the resources. But that's a declaration of bankruptcy for the whole "Linux is built by community volunteers" thing.
May well be the case.
Somewhere GregKH mentioned how the pace of kernel development had changed, with git being a major part of it.
It may well be that the pace have now gotten to the point where hobbyists have a hard time keeping up, as they have things they need to do besides stare at code all day.
This has been the case for some time now, but the ecosystem of large-ish companies was diverse enough as to avoid appearing dominated by this or that player. The fall of Novell and Nokia, coupled with Ubuntu's various pivots and IBM's troubles, left Red Hat as the lone real force with the capability to steer the most significant projects.
This is nobody's fault, but it's not healthy in the long run.
> It may well be that the pace have now gotten to the point where hobbyists have a hard time keeping up, as they have things they need to do besides stare at code all day.
Totally agree. The needs of modern computing (server-, desktop- and security-wise) have drastically increased IMHO. You need to read so much documentation and existing code before you can even start coding. That's just not feasible in your free time after work if you're having an active social life or another computer-unrelated hobby.
Although I have to admit I might be biased about this topic. I'm contributing to a FOSS project which got a lot of flak for trying to raise money for employees in a ... well, more aggressive way. While not necessarily agreeing with everything that happened, I do believe you need to work full-time on modern FOSS projects to push ahead.
This is Too Big To Fork https://news.ycombinator.com/item?id=6810259 at work. Any legally free-slash-open-source software project which is so complex that only a few big actors have the will and ability to make and maintain a fork is de facto under the shared proprietary control of those big actors. (The same goes for software where control of the installed base means de facto control over any changes to widely-used interfaces.) "Freedom of the press is guaranteed only to those who own one." The Linux 'ecosystem' is only unusual in that (as it seems) there's exactly one big actor left standing by now.
> Debian isn't really a leader. They're more of a passive target platform and their committee has people from various strokes of the Linux community.
This is so not true that it is bordering on FUD. Systemd's leadership depended on debian making a fairly democratic, open and violently fought battle between upstart vs systemd. Everyone knew that Ubuntu (which is pretty much defacto installed in every laptop sold in Asia) would adopt systemd based on Debian's decision.
Not only did Debian make the decision to support systemd, it voted to NOT support other init systems. Mark Shuttleworth made his announcement the day after (http://www.markshuttleworth.com/archives/1316 ).
It is an interesting position to take by painting systemd as the lackey of a capitalist monopoly trying to take over the world. I can see how that can get a lot of mindshare. The truth is far simpler - systemd is far superior.
RH is the 800-pound gorilla of the Linux world. They employ a sizeable portion of the developers working on various projects within the Linux ecosystem.
Canonical revenue is 67 million according to wikipedia, by contrast Red Hat is 1.5 billion. That is 3 extra zeros (and even then it is almost 1/100 of Microsoft's revenue).
Or at least don't clash. Intel is mostly involved on the hardware end. The biggest clash is perhaps Oracle, as they forked RHEL some years back. but i think that is more about shoring up a silo around their database business, rather than getting involved in the general workstation and server business.
But that is a recent move, while RH has been in the Linux development effort for some time.
At this point RedHat has consumed de-facto governance of some core projects that make up the Linux desktop system that sit atop the kernel. They pay developers to work on these projects and there employees have decision making authority. SystemD is almost entirely driven by current/former RedHat employees.
The fact of the matter is that "Distros" like Debian etc are dependent on upstream developers to provide new versions of there system. If a large portion of important subsystems are developed by RedHat developers then they will adopt that code and be driven in the direction upstream wants to go.
"Distros" do not develop, they package upstream into compiled binaries and add there configs and perhaps package management. In short most of the distros will bend which ever way the upstream wind blows.
Thats why I use a "Distro" Like Slackware or Crux that is built from scratch and not based on another system. What limited autonomy there is in Linux land is with the independent Distros.
Ultimately you can go for something like Linux From Scratch. And even they have gotten somewhat fed up with the Systemd antic.
For instance their main book use eudev rather than udev, because they found the effort of extracting udev from the larger systemd project a right pain.
They do however maintain a parallel systemd book for anyone interested.
Linux was/is an extremely flexible system. The vendors like RedHat should be able to refashion Linux in what ever way they see fit. The big problem with SystemD is that it stepped over the red line from in-house RedHat Linux component like SE-Linux and there various "enhancement" to a rigid default that attempts to lock down a large array of system components under a project that has commercial motives and drivers, driven by a single vendor. And it was done with a pre-meditated social engineering push which was quite nasty. Big no-no for Linux and Open Source eco-system in general. Projects Like Linux from scratch, GNU, Busybox etc will not role over for the endless attempts at vendor lockin. Its all been tried before and while the players may be new the game is very old and ultimately we will defeat these new attempts just like we did with SCO and the proprietary Unix/Microsoft Corporations that tried to kill Linux in the 90s.
By comparison Debian just has enough resources for keeping Debian running. SUSE seems to be an afterthought at this point. Ubuntu seems to mostly be concerned with itself.
If ARM became a big platform for servers or workstations, RH would probably roll out a ARM variant within the week (could be they even have test versions floating around their internal network).
In particular as Fedora already have ARM versions across the board.
My comment was more the fact that Debian has enough excess resources to support the Beaglebone Black. So, categorizing Debian as having barely enough developers to support itself is disingenuous.
Nonetheless, Red Hat really is doing a lot of upstream development, certainly more than Debian does as far as I know. Indeed, the Debian community has managed to keep their release schedule and meet their goals lately, so they're probably in a better shape than "barely enough to support itself", but there hardly seems to be room for comparison between the two.
In this case the decision does influence other people, because as pointed out, systemd has taken over many other components' roles, not just init, and if those components are no longer maintained because RedHat pulls its support, then other distributions (with far far fewer resources) are forced to use unmaintained code or switch to systemd.
Red Hat is definitely under no obligations to maintain things it doesn't see as a valuable, and if no one else wants to pick up the slack either then action and code speaks much more loudly then words about the actual value people assign those things.
I'm just guessing here, but systemd makes some things more convinient by being overall more monolithic, like Windows, and especially distros that are targeting a less technical crowd want to be able to compete with Windows in terms of features and usability. Non-technical people most likely don't see any of the beauty of a strictly modular system or proper clean code.
And this is not about the init-daemon, it is about systemd as a whole integrated suite of daemons that connect easily and add fancy new features, while on the other side you might have to modify two completely independent software projects to add a new feature that uses both, and even then there most likely are alternatives to each one, and you feature will only work with this specific combination. So a more monolithic system, which systemd is despite Lennard denying this, makes development faster, but will bite you in the long run, especially when the code and documentation quality is as poor as systemd's.
I'm more tempted to say that systemd does well what previously you needed a dozen half-broken tools to implement.
For example, isolating the /tmp of an application, monitoring the process and restarting if necessary, more logging options (and imho cleaner/more efficient).
Then again, all this has been debated over and over again. In practice, people are adopting it. Haters are noisy. It's like reading about debates on IPv6 migration strategies.
Process supervisors with reliable logging are nearly two decades old at this point. Isolating /tmp is then a namespacing feature, which in a system where execution state is composed as an explicit external manifest via chain loading (as opposed to serializing from a unit file into a private ExecContext structure, as with systemd) should be completely orthogonal to any one service manager. Else it is inflexible if it needs complex internal scaffolding.
(Actually I think supervision goes back to IBM's SRC in at least 1992, but I'm not exactly sure if the earliest versions had anything beyond process management.)
I think one thing that gave systemd adoption a big boost was a declaration from the maintainer of the cgroups sub-system of Linux.
At present multiple processes can set up and manage cgroups, but the maintainer wants to change to there being a single user space manager.
Thus systemd was pushed as "the" cgroups user space process.
You can already seeing the systemd devs behaving as if this was a done deal.
Ran into a email a while back about systemd clobbering a libvirt managed cgroup. And in it Poettering "suggested" that libvirt should hand control over to systemd as it would be THE cgroups manager going forward.
You can probably find similar encounters between, say, Docker and systemd.
Which is a failure of people making server distros, not those writing process supervisors.
Not at all. SMF supports delegated restarters, its milestones can contain far more state information (and explicitly) than targets can, it uses a configuration repository which enables runtime dynamic service state modification at a far higher rate than the relatively static systemd, its dependency system is simpler, it's integrated with the hardware fault manager (also to provide service resource identifiers beyond the COMM name), its profiles are more granular than presets, it has explicit service instance support and logging is flat file-based in /var/svc/log.
SMF has plenty of its own issues, but it's quite different from systemd. I think the delegation aspect is one of its most valuable lessons, but ultimately I also believe it's too overarching for a modern init.
Well that's a good question, I'd like to have an answer to that.
debian adoption of systemd was a bumpy ride to say the least, it caused a few long time contributors to resign and others to fork debian to remove systemd in a new distro called devuan.
Then again systemd gobbled other critical components such as udev, there's also gnome that made it a strict requirement, like a cancer it grows and takes over other components.
Devuan is someone's extended tantrum and little more. Like every other rage-fork it'll die a slow death because who wants to develop on a platform founded on the premise of "why do we need to change anything? It's all working fine!"
They began work on a logind compatibility layer over ConsoleKit2, they're writing a NetworkManager alternative, they directly influenced and are supporting a udev alternative called vdev (which also has libudev compatibility), and a host of other things.
For a rage-fork, it's pretty impressive. They're changing a lot. It's easier to just astroturf in the corner, though, I suppose.
Possibly because vdev is a clean break, while eudev is still mostly about udev stripped from systemd.
And that stripping will be more complicated moving forward, as i recall a recent systemd release moved various bits from udev to a new systemd lib. Leaving the udev interfaces as stubs to be removed at some undetermined future date.
Yeah the most documented such evaluation, the Debian process, seemed to only evaluate sysv, upstart and systemd (openrc was briefly mentioned but quickly dismissed).
The rest seems to have been executive decisions (with or without a "deal with it" meme accompanying), often by people already involved with systemd development.
I haven't evaluated all of the previous attempts since that would consume more time than I have available. Can you provide a comparison or some insights into why they're worse?
If you get time that would be neat, I'm sure others would appreciate it too. I've heard good things about djb's daemontools from a few friends and colleagues but never had the chance to try it and if you have any insight on that one I'd really appreciate that.
I'd also like an answer to this question, as well as an answer to the question of what, in exacting detail please, was so wrong with the previous system it needed to be torn out and replaced?
I definitely tend toward the curmudgeonly, but to this grumpy old man, it seems like we're replacing things simply for the sake of change.
it did, but systemd is not the silver bullet that will fix it.
"It's often said that a half-truth is worse than a lie. In the case of systemd, a half-improvement is worse than a complete flop. People are focusing on the features they want to use and ignoring the ones that will come back to bite them later.
No, my biggest problem with systemd is that it is repeating the mistakes of System V all over again. It's just given them a change of clothes to appeal to the 21st century."
At this point in time though i wonder if focusing on the init part is missing the forest for the trees.
I think just as much of a stink comes with how you need to have systemd-the-init to use systemd-logind (replacing consolekit for session tracking) or any number of other possibly interesting, but tied to the hip of systemd, sub-projects.
Arch linux spawned a distro called manjaro which leads the way on not joining the systemd bandwagon. Debian got forked into devuan which is debian minus systemd. As systemd originated from redhat it is expected that the other faces of redhat namely fedora and centos would also feature systemd.
Then there is slackware, gentoo, pclinuxos that either did not bit the systemd bullet or offer alternative option. Not sure about ubuntu.
"Arch linux spawned a distro called manjaro which leads the way on not joining the systemd bandwagon."
...well, kinda: manjaro supports openrc, but the official, main installs are systemd. you can start with an official install and convert it, or you can use an openrc iso, but those are not officially supported.
Just to add to the others CentOS was essentially bought by RedHat and there Core developer is now a RedHat employee. Magically there was a new major 7.0 version bump adopting SystemD as the one and only init system. A new flashy website and a refusal to allow any alternative init system into the official packages other then SystemD. Needles to say CentOS is no longer an independent alternative to RedHat Linux.
RedHat has to release there sources for others to use. Most of the code is under some form of GPL. CentOS was an independent volunteer driven effort that compiled and packaged that code into a new distro not affiliated with RedHat. That is why they had to remove RedHat branding and they could not market CentOS as RHEL etc.
Can we get a bit a of context on that?