Skip to Content [alt-c]

Andrew Ayer

Sections

How to Crash Systemd in One Tweet

The following command, when run as any user, will crash systemd:

NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""

After running this command, PID 1 is hung in the pause system call. You can no longer start and stop daemons. inetd-style services no longer accept connections. You cannot cleanly reboot the system. The system feels generally unstable (e.g. ssh and su hang for 30 seconds since systemd is now integrated with the login system). All of this can be caused by a command that's short enough to fit in a Tweet.

Edit (2016-09-28 21:34): Some people can only reproduce if they wrap the command in a while true loop. Yay non-determinism!

The bug is remarkably banal. The above systemd-notify command sends a zero-length message to the world-accessible UNIX domain socket located at /run/systemd/notify. PID 1 receives the message and fails an assertion that the message length is greater than zero. Despite the banality, the bug is serious, as it allows any local user to trivially perform a denial-of-service attack against a critical system component.

The immediate question raised by this bug is what kind of quality assurance process would allow such a simple bug to exist for over two years (it was introduced in systemd 209). Isn't the empty string an obvious test case? One would hope that PID 1, the most important userspace process, would have better quality assurance than this. Unfortunately, it seems that crashes of PID 1 are not unusual, as a quick glance through the systemd commit log reveals commit messages such as:

Systemd's problems run far deeper than this one bug. Systemd is defective by design. Writing bug-free software is extremely difficult. Even good programmers would inevitably introduce bugs into a project of the scale and complexity of systemd. However, good programmers recognize the difficulty of writing bug-free software and understand the importance of designing software in a way that minimizes the likelihood of bugs or at least reduces their impact. The systemd developers understand none of this, opting to cram an enormous amount of unnecessary complexity into PID 1, which runs as root and is written in a memory-unsafe language.

Some degree of complexity is to be expected, as systemd provides a number of useful and compelling features (although they did not invent them; they were just the first to aggressively market them). Whether or not systemd has made the right trade-off between features and complexity is a matter of debate. What is not debatable is that systemd's complexity does not belong in PID 1. As Rich Felker explained, the only job of PID 1 is to execute the real init system and reap zombies. Furthermore, the real init system, even when running as a non-PID 1 process, should be structured in a modular way such that a failure in one of the riskier components does not bring down the more critical components. For instance, a failure in the daemon management code should not prevent the system from being cleanly rebooted.

In particular, any code that accepts messages from untrustworthy sources like systemd-notify should run in a dedicated process as a unprivileged user. The unprivileged process parses and validates messages before passing them along to the privileged process. This is called privilege separation and has been a best practice in security-aware software for over a decade. Systemd, by contrast, does text parsing on messages from untrusted sources, in C, running as root in PID 1. If you think systemd doesn't need privilege separation because it only parses messages from local users, keep in mind that in the Internet era, local attacks tend to acquire remote vectors. Consider Shellshock, or the presentation at this year's systemd conference which is titled "Talking to systemd from a Web Browser."

Systemd's "we don't make mistakes" attitude towards security can be seen in other places, such as this code from the main() function of PID 1:

/* Disable the umask logic */
if (getpid() == 1)
        umask(0);

Setting a umask of 0 means that, by default, any file created by systemd will be world-readable and -writable. Systemd defines a macro called RUN_WITH_UMASK which is used to temporarily set a more restrictive umask when systemd needs to create a file with different permissions. This is backwards. The default umask should be restrictive, so forgetting to change the umask when creating a file would result in a file that obviously doesn't work. This is called fail-safe design. Instead systemd is fail-open, so forgetting to change the umask (which has already happened twice) creates a file that works but is a potential security vulnerability.

The Linux ecosystem has fallen behind other operating systems in writing secure and robust software. While Microsoft was hardening Windows and Apple was developing iOS, open source software became complacent. However, I see improvement on the horizon. Heartbleed and Shellshock were wake-up calls that have led to increased scrutiny of open source software. Go and Rust are compelling, safe languages for writing the type of systems software that has traditionally been written in C. Systemd is dangerous not only because it is introducing hundreds of thousands of lines of complex C code without any regard to longstanding security practices like privilege separation or fail-safe design, but because it is setting itself up to be irreplaceable. Systemd is far more than an init system: it is becoming a secondary operating system kernel, providing a log server, a device manager, a container manager, a login manager, a DHCP client, a DNS resolver, and an NTP client. These services are largely interdependent and provide non-standard interfaces for other applications to use. This makes any one component of systemd hard to replace, which will prevent more secure alternatives from gaining adoption in the future.

Consider systemd's DNS resolver. DNS is a complicated, security-sensitive protocol. In August 2014, Lennart Poettering declared that "systemd-resolved is now a pretty complete caching DNS and LLMNR stub resolver." In reality, systemd-resolved failed to implement any of the documented best practices to protect against DNS cache poisoning. It was vulnerable to Dan Kaminsky's cache poisoning attack which was fixed in every other DNS server during a massive coordinated response in 2008 (and which had been fixed in djbdns in 1999). Although systemd doesn't force you to use systemd-resolved, it exposes a non-standard interface over DBUS which they encourage applications to use instead of the standard DNS protocol over port 53. If applications follow this recommendation, it will become impossible to replace systemd-resolved with a more secure DNS resolver, unless that DNS resolver opts to emulate systemd's non-standard DBUS API.

It is not too late to stop this. Although almost every Linux distribution now uses systemd for their init system, init was a soft target for systemd because the systems they replaced were so bad. That's not true for the other services which systemd is trying to replace such as network management, DNS, and NTP. Systemd offers very few compelling features over existing implementations, but does carry a large amount of risk. If you're a system administrator, resist the replacement of existing services and hold out for replacements that are more secure. If you're an application developer, do not use systemd's non-standard interfaces. There will be better alternatives in the future that are more secure than what we have now. But adopting them will only be possible if systemd has not destroyed the modularity and standards-compliance that make innovation possible.

Hi, I'm Andrew. I'm the founder of SSLMate, a service which automates your SSL certificate deployment. I also develop open source projects like git-crypt and titus.

I blog here about a variety of technology topics, including security, devops, IPv6, and reliable programming. If you liked this post, check out my other posts or subscribe to my Atom feed.

My email address is andrew@agwa.name. I'm AGWA at GitHub and @__agwa on Twitter.

Comments

The comments below are owned by whoever posted them. I am not responsible for them in any way.

i just ran it. nothing happened. everything works exactly the same as before

| Posted on 2016-09-28 at 20:33:07 UTC by Reader me | Reply to This

Some people are having to run the command in a loop for the bug to trigger: https://github.com/systemd/systemd/issues/4234#issuecomment-250289253

I'm not sure why; I was able to consistently reproduce with a single call under systemd v215, v229, and v230, on Debian and Ubuntu.

| Posted on 2016-09-28 at 21:32:28 UTC by Andrew Ayer | Reply to This

Thank you for providing tangible arguments as to why Systemd is a dangerous option.

| Posted on 2016-09-28 at 21:09:27 UTC by Anonymous | Reply to This

Please email them on secalert <@> redhat.com and they will take your issue seriously.

| Posted on 2016-09-29 at 00:13:40 UTC by Reader Anonymous | Reply to This

maybe it would have been a good idea to send that email yourself rather then advising someone else to do so. ... and 'yes' I did it now

| Posted on 2016-10-02 at 21:33:41 UTC by Anonymous | Reply to This

from secalert@redhat.com

Thank you for this notification. We are aware of this systemd issue and it is being tracked in this flaw:

https://bugzilla.redhat.com/show_bug.cgi?id=1380286

| Posted on 2016-10-04 at 00:11:43 UTC by Anonymous | Reply to This

I found I had to use a "while true" loop around that before it would trigger, as it seems it doesn't reliably trigger with every attempt (curious what combination of circumstances trigger it?)

Just out of curiosity, why didn't you contact the RedHat security team to responsibly disclose this vulnerability? (the majority of systemd development work is funded by, and done by, RedHat employees)

| Posted on 2016-09-28 at 21:09:44 UTC by Reader Twirrim | Reply to This

This article starts with a bug. Note, it's a pretty irrelevant one that doesn't appear to be reproducible on recent builds.

Then it brings up a few other debatable issues, and then it incorrectly generalizes that systemd security is terrible. I say debatable because for example parsing command line parameters isn't something people usually delegate to a separate process.

Unfortunately, it also ignores all the great systemd security features it has added for the average Linux user (such as private tmp, and cgroups), and in fact all the other great features it has.

| Posted on 2016-09-28 at 22:00:54 UTC by Reader Keith Curtis | Reply to This

parsing command line parameters isn't something people usually delegate to a separate process.

It isn't about parsing command line. It's about parsing text coming from an untrusted socket where anyone can write to, at runtime, in something that runs as PID 1. Not exactly the same thing.

Unfortunately, it also ignores all the great systemd security features it has added for the average Linux user (such as private tmp, and cgroups), and in fact all the other great features it has.

What it's debatable is that systemd add those things. I use private-tmp, namespaces, cgroups and more... and i don't have systemd installed in my system.

And anyway, cgroups are NOT a security feature; they can help to get a better security, but they aren't a security measure.

| Posted on 2016-09-29 at 00:44:19 UTC by Reader nextime | Reply to This

Private tmp (being one application of namespaces) and cgroups are features of the Linux kernel, not systemd.

Also, the bug is very reproducible. I just tried it in Ubuntu 16.04 LTS after apt upgrade. Worked very, well^Wbadly.

| Posted on 2016-09-29 at 07:22:22 UTC by Reader Antti Laine | Reply to This

I couldn't reproduce it on Ubuntu 16.04 LTS 32 bit.

| Posted on 2016-09-30 at 08:33:17 UTC by Anonymous | Reply to This

Ubuntu has released a security update that fixes this bug: http://www.ubuntu.com/usn/usn-3094-1/

However, it introduced a new, different DoS vulnerability (file descriptor leak) so expect a second security update soon.

| Posted on 2016-09-30 at 15:35:27 UTC by Andrew Ayer | Reply to This

Um... I can repeat it on 7 fully patched. I'm not sure what you mean by irrelevant. Now the patch has been submitted. When will a CVE be released. Sticking our heads in the sand is so ... Microsofty .... Let's not follow the win98 crew on that aspect please.

| Posted on 2016-09-29 at 19:58:35 UTC by Reader James | Reply to This

Why not report this kind of denial-of-service attack in a responsible disclosure manner?

| Posted on 2016-09-28 at 23:12:05 UTC by Reader Stefan | Reply to This

and you know that it was not responsibly reported how?

| Posted on 2016-09-29 at 13:56:29 UTC by Reader tim | Reply to This

Where is the bug report then?

| Posted on 2016-10-01 at 05:26:11 UTC by Reader Simon Strandman | Reply to This

He wouldn't have gotten clicks for his blog if he had.

See, nowadays assholes like the OP think they're contributing something but in reality they're just trying to make enough so someone will share their blog.

Hits > Actually doing something worthwhile.

| Posted on 2016-10-02 at 13:51:22 UTC by Anonymous | Reply to This

Making the world aware of the sort of thing systemd folks think is okay is the most responsible thing one can do.

Everyone knew systemd was a mess. Everyone knew they were going to try to take over the world. Everyone knew it was bound to have problems like this. They were warned many times already, and chose to go with Lennart's handwaving anyway.

| Posted on 2016-10-02 at 19:30:20 UTC by Anonymous | Reply to This

Well, (1) Thanks for showing this error, it is just so...so... so... "expected?"

Yes, "everybody" knows it a bad idea, but then why follow it, 'cause it was solving all the "right" problems, in the wrong way, and everybody was looking at the marketing babble and just wanted the "pains" relieved, instead of understanding their souls their are selling out ;( It sorta started with Upstart, and was unavoidable that something will come in, but then, as is always said in OpenSource: Code talks harder than words...

Wish I had the budget and team to write a "proper" replacements similar to Solaris 10/11's service management framework, but in user mode.

| Posted on 2016-10-06 at 11:15:52 UTC by Reader Hendrik Visage | Reply to This

this is the most comprehensive writeup I've yet seen illustrating why systemd is not just a bad idea, but a worse implementation. The very concept is the complete opposite of the past 40 years of UNIX wisdom; that so many (well-respected) systems design folk seem to have completely forgotten their history (and really, just common sense) is depressing indeed.

| Posted on 2016-09-29 at 00:29:25 UTC by Anonymous | Reply to This

(intended to sign that post, doing so here.)

| Posted on 2016-09-29 at 00:30:01 UTC by Reader Scott Francis | Reply to This

http://static.usenix.org/event/lisa05/tech/full_papers/adams/adams.pdf This must be what systemd people saw but failed to implement in sane way.

| Posted on 2016-09-29 at 07:02:00 UTC by Reader Jussi Sallinen | Reply to This

Fully agree. I'm just wondering what will this time be the excuse for closing the issue/bug report without even admitting the bug exists, or that it is a bug.

| Posted on 2016-09-29 at 10:43:40 UTC by Reader Matthias Koch | Reply to This

They fixed the bug (although it took them two tries to get it right). But it's not about the bug. The bug is just one example of systemd's terrible programming practices.

| Posted on 2016-09-29 at 23:18:07 UTC by Andrew Ayer | Reply to This

Do you expect any software to be bug free from day one or just systemd? the sweeping generalisations you make are really stupid

| Posted on 2016-10-01 at 17:23:39 UTC by Reader rtfa | Reply to This

Do you expect any software to be bug free from day one or just systemd?

This is day 2,337 for systemd. I'm not the OP, but I expect that system-critical system management software gets input sanitization and separations of concerns correct... especially six years after its initial public release.

| Posted on 2016-10-02 at 05:35:10 UTC by Reader anonnymoose | Reply to This

The Unix philosophy is that you do one thing and to do it well.

That means that you should have a program reaping zombies and do nothing else.

You should have a program that starts the system services taking their dependencies into account. The oldfashioned way is to do them in a specific order, the new way is to calculate the dependencies. You should have a simple program that asks for an IP address on a network interface and does nothing else but to assign that IP address on that interface.

All these are simple programs that can be reasonably comprehended and verified.

Once you integrate all this, the complexity increases, and the amount of work to maintain the code base increases super-linearly. And as agwa says: the consequences of a problem become more troubling. Use the tools that are already there. The "runlevel switching" may want to accept "requests" from userspace. Make that a root-only socket or something.

If you want to make users able to switch runlevels or other "normally root" stuff, delegate that to "sudo" or modern variants of such tools. Or make a daemon that will accept commands from the normal users and passes them on as a privileged one. Again a bug somewhere in this process has the problems contained.

| Posted on 2016-10-02 at 13:23:12 UTC by Anonymous | Reply to This

From my previous years in commercial software engineering. "Closed - Functions as intended."

| Posted on 2016-10-01 at 23:49:12 UTC by Anonymous | Reply to This

I agree with a large part of this. The only things I disagree about:

1. As a user and hobbyist system admin, systemd has numerous advantages over sysv init(which was the only previous init system to be installed by default on most distributions). So I would not say it has "very few" advantages over other init systems (I haven't used the other more modern init systems enough to have an informed opinion there). I also think having one standard init system between most distributions is very nice, so the fragmentation of the previous non-sysv solutions annoyed me.

2. Systemd, by definition, does not use a "proprietary" DBUS API nor "proprietary" interfaces. See http://www.merriam-webster.com/dictionary/proprietary if you don't believe me. Anyone can legally and practically implement a systemd DNS service clone that communicates via d-bus exactly like the systemd one, if they wish. That being said, I agree that it can still be a problem since it forces others to either imitate systemd's interface or get applications to use something standardized instead/as well.

| Posted on 2016-09-29 at 01:03:36 UTC by Reader Christopher W. Carpenter | Reply to This

1. You misread me. I acknowledge that systemd's init system functionality has compelling features, even if terribly implemented. It's the other functionality -- DNS, DHCP, NTP -- where it doesn't offer anything new.

2. Fair enough. I replaced "proprietary" with "non-standard," which is what I meant.

| Posted on 2016-09-29 at 01:56:44 UTC by Andrew Ayer | Reply to This

As a user and hobbyist system admin

You seem to be a better adapted sysadmin than most of the non-hobbyist sysadmin's I know!

| Posted on 2016-09-29 at 02:07:32 UTC by Reader jampola | Reply to This

I am a user and a professional Systems Admin. I might well note that this is the very kind of thing that does nothing for me. And a whole lot to me. I'd like to spend less time working around the things it breaks, please.

| Posted on 2016-09-29 at 20:00:57 UTC by Reader James | Reply to This

This article is fantastic. I've been screaming about the rotten state of Linux due to Poettewrong's sysd for months. I've moved everything over to devuan, or left wheezy in place. Oh and I like sysvinit. So there. I REALLY like having log files I can tail too.

| Posted on 2016-09-29 at 05:21:37 UTC by Reader Rhy | Reply to This

Ok Andrew, it works, i tested it in Debian Stable and Ubuntu LTS 16.04, every systemd distro that relies all their init scripts in systemd will crash with that command with an unprivileged user, so my Debian Stable with old legacy init scripts is not vulnerable, new Ubuntu systemdized distros (16.04) will crash systemd, no new processes can be launched, launched processes can't restart/reload, you can't even shutdown or reboot gracefully... It's amazing but what about Responsible Disclosure?

| Posted on 2016-09-29 at 08:59:08 UTC by Reader ferchunix | Reply to This

Slackware is not using this bullshit of systemd and working fine for many years now !

| Posted on 2016-09-29 at 09:52:56 UTC by Reader freesys59 | Reply to This

Devuan (https://devuan.org/) is Debian 8 systemd-free

| Posted on 2016-09-29 at 12:17:46 UTC by Reader Andrea Mistrali | Reply to This

This does not appear to apply to OpenSUSE 13.2, as the path /run/systemd/notify does not exist. Running the command systemd-notify "" on its own does not appear to trigger the behaviour.

I'm not sure why that path doesn't exist.

| Posted on 2016-09-29 at 10:42:03 UTC by Reader Wolf | Reply to This

Can you provide a link to the bug report you submitted to the systemd team before writing this blog post?

| Posted on 2016-09-29 at 10:53:26 UTC by Reader James J. | Reply to This

People don't seem to understand how assert works. In release builds assert should be no-op. https://github.com/lattera/glibc/blob/master/assert/assert.h#L44 First of all assert is a debug tool, not error handling mechanism. Secondly, exiting with abort signal isn't very good error handling mechanism for systemd.

On the other hand, arguing that this makes systemd defective is hypocritical considering that Go FAQ says

Go doesn't provide assertions. They are undeniably convenient, but our experience has been that programmers use them as a crutch to avoid thinking about proper error handling and reporting. (https://golang.org/doc/faq#assertions)

| Posted on 2016-09-29 at 11:45:12 UTC by Reader Juha Autero | Reply to This

systemd's assertions are enabled in release builds. However, systemd doesn't exit when an assertion fails. Instead it hangs in the pause system call. This prevents the whole system from crashing, although the system is left in a degraded state.

I would have to say that this bug supports Golang's position on assertions.

| Posted on 2016-09-29 at 23:21:15 UTC by Andrew Ayer | Reply to This

:(){ :|: & };:

| Posted on 2016-09-29 at 12:23:17 UTC by Anonymous | Reply to This

Handled by age old unix configurations long before systemd came to be...

| Posted on 2016-10-09 at 13:46:43 UTC by Anonymous | Reply to This

Nobody I know likes systemd, it's too intrusive in user space and goes way against the core Unix philosophy of doing one thing and doing it well.

The sooner systemd disappears out of the Unix/Linux eco system the better.

Thank you for exposing this bug. We tried it on our up-to-date RedHat 7.2 systems and sure enough they crashed very hard.

Now to get rid of Not-workManager.

| Posted on 2016-09-29 at 12:25:17 UTC by Reader orlwrite | Reply to This

Shrike?!

| Posted on 2016-10-12 at 18:10:56 UTC by Anonymous | Reply to This

If you haven't read it, check out this issue for another questionable security decision:

https://github.com/systemd/systemd/issues/959

| Posted on 2016-09-29 at 14:29:22 UTC by Reader gdfuego | Reply to This

You could think systemd is as dangerous as ISIS

| Posted on 2016-09-29 at 16:31:16 UTC by Anonymous | Reply to This

I feel sorry for people using distributions with systemd ''baked in''. During my day job, I am actually one of them, since my place of work is a big user of either Red Hat or Centos.

And then I remember that my personal systems are running either BSD or Slackware. And I feel better about my life.

| Posted on 2016-09-29 at 16:55:45 UTC by Reader Noryungi | Reply to This

In some parts of SystemD they seem to use "bad coding practices from textbook":

} else if (startswith(option, "keyfile-offset=")) { if (safe_atou(option+15, &arg_keyfile_offset) < 0) { log_error("keyfile-offset= parse failure, ignoring."); return 0; }

Do you see "option+15"? "15" here is a magic number, denoting the length of "keyfile-offset=":

https://cgit.freedesktop.org/systemd/systemd/tree/src/cryptsetup/cryptsetup.c#n106

| Posted on 2016-09-29 at 18:38:50 UTC by Reader Andrey Bergman | Reply to This

Send patch?

| Posted on 2016-09-29 at 21:37:24 UTC by Reader Damjan | Reply to This

I don't use systemD, so I even can't test the patch.

| Posted on 2016-09-30 at 02:52:56 UTC by Reader Andrey Bergman | Reply to This

So what's the intention to post this code example on various places (like Golem.de)? You don't use systemd, that's fine! So show us your code - maybe we can learn from you!

| Posted on 2016-10-01 at 11:58:51 UTC by Reader Stefan | Reply to This

Dear Stephan, you are very welcome to learn from me. This code snapshot with my comment teaches you that including magic numbers in the code is considered a bad practice for half a century.

| Posted on 2016-10-01 at 16:27:29 UTC by Reader Andrey Bergman | Reply to This

nobody uses systemD, linux basesystem is called systemd you can't test bad coding practices either, but it did not stop you

| Posted on 2016-10-01 at 14:19:49 UTC by Anonymous | Reply to This

The linux "basesystem" is called the "kernel" and is normally named vmlinuz. systemd IS NOT the BASESYSTEM of ANY LINUX distro - it is a service manager - which used to handled by initd - which only did one thing - init processes. It IS possible to TEST for magic numbers as well as review code. Good coding principles: 1. DRY - Don't repeat yourself - strings should exist in one place and be declared const. The length of the string if used often should be should be recorded as a singleton value also marked const at initialization. 2. Initialization is Instantiation. Goes without saying. There should not be ANY arithmatic with static numbers when doing text parsing. +15 and +XYZ is the exact same number of instructions in assembly. There is no code savings or clock cycle savings.

Learn to speak intelligently before you post.

| Posted on 2016-10-02 at 06:45:38 UTC by Anonymous | Reply to This

So feel free to write a bug report or even send a patch (or discuss on systemd-devel).

Otherwise it is damn useless to write comments...

| Posted on 2016-10-02 at 08:44:44 UTC by Reader Stefan | Reply to This

Yeah, it is totally useless if the other party refuses to listen and the only thing he can do is bawling like a baby. That's what you should't do as a developer, go and fix your damn spaghetti code!

| Posted on 2016-10-02 at 12:52:49 UTC by Anonymous | Reply to This

Yeah right, reporting bugs to poettering? "closed, not a bug" "closed, works as designed" "closed, i am right and you know nothing" "closed, everything else is crap and systemd does it the right way"

| Posted on 2016-11-06 at 04:37:23 UTC by Anonymous | Reply to This

I continue not to understand why so many high-profile distributions' Release Configuration Managers drank the Flavor-Aid.

And the bad part is not that they did, but that because they did, packages are no longer required to have initscripts for sane inits, meaning that you have no choice: you either run systemd (read: shoot yourself in the head), or you build initscripts by hand for every package you want to run.

The only thing worse is reading the replies from systemd partisans when it's explained to them why the choices systemd's design makes are bad engineering practice -- am I the only one who thinks the structure of these replies is way too similar to the replies one hears from Trump supporters when it's explained why he is bad engineering practice? :-)

Added this article to my list of "Why systemd is bad" articles; added the supporting commenters to my list of "people never to hire or recommend for responsible software engineering jobs".

| Posted on 2016-09-29 at 18:48:13 UTC by Reader Baylink | Reply to This

Gnome and developer hours available basically.

With Gnome yanking out consolekit support in favor of logind, and logind being part of the system package (not only is everything in the same git, they are all in the same src tree! When the LFS team shifts to eudev because extracting udev from the larger systemd source ball is too much effort you know something is up.) not moving to systemd becomes a hard sell.

Frankly the whole consolekit/logind thing seems overdesigned and aimed squarely at reproducing Microsoft's Active Directory. Is some corporate admin wants that, let him enable it. There is crap all reason for shipping this on every distro unless you think every last of your users are idiots (oh wait, Gnomes already do...).

| Posted on 2016-10-01 at 20:48:12 UTC by Anonymous | Reply to This

wouldn’t it be enough to have a simple syntax which can do everything systemd init syntax can?

Something like this?

MOUNTPOINT=/sys/fs/fuse/connections depend() { need localmount } start() { ebegin "Starting fuse" if ! grep -qw fuse /proc/filesystems; then modprobe fuse >/dev/null 2>&1 || eerror $? "Error loading fuse module" fi if grep -qw fusectl /proc/filesystems && \ ! grep -qw $MOUNTPOINT /proc/mounts; then mount -t fusectl none $MOUNTPOINT >/dev/null 2>&1 || \ eerror $? "Error mounting control filesystem" fi eend ${?} } stop() { ebegin "Stopping fuse" if grep -qw $MOUNTPOINT /proc/mounts; then umount $MOUNTPOINT >/dev/null 2>&1 || \ eerror $? "Error unmounting control filesystem" fi eend ${?} }

That’s the syntax of OpenRC. You’ll notice the declarative dependencies along with the simple scripting in start() and stop().

And since it’s a scripting language, the answer to "can it do everything systemd can" is simply: Yes, it can.

A simple standard along with some convenience functions and building on existing tools already provides most of the functionality of systemd.

| Posted on 2016-10-06 at 08:47:52 UTC by Anonymous | Reply to This

At $DAY_JOB we have just moved our entire stack to Alpine, which features the wonderful OpenRC :)

| Posted on 2016-10-12 at 18:14:47 UTC by Anonymous | Reply to This

It's only because of the declarative syntax of systemd. If we could reverse engineer the systemd syntax with something more stable, we could go back to business as usual. Init scripts are hard to write. Writing configurations that are standard are easy. Systemd just happens to be bad code around a good idea - declarative definitions of services. However, trying to be everything and do everything is causing the bigger problems.

My system hangs because it takes just a little longer for all my items to come online. And that in turns brings 32GB, Core-I7 4720HQ to its knees on a reboot. I always have to restart the display-manager because too many things are starting at once.

| Posted on 2016-10-02 at 06:48:33 UTC by Anonymous | Reply to This

Good luck with that, as they seem to pile on new declarations with every release.

There is one guy that is trying though.

http://jdebp.eu./Softwares/nosh/

| Posted on 2016-10-02 at 07:19:55 UTC by Anonymous | Reply to This

Comment 31898 should have been a reply to you. Sorry: https://www.agwa.name/blog/post/how_to_crash_systemd_in_one_tweet#comment-31898

| Posted on 2016-10-06 at 08:49:14 UTC by Anonymous | Reply to This

FUCK systemd!!!

| Posted on 2016-09-29 at 22:43:18 UTC by Reader Linus | Reply to This

Does not work.

It say: "Failed to notify init system"...

| Posted on 2016-09-30 at 07:54:21 UTC by Reader funt | Reply to This

That probably means systemd has already crashed. Try using systemctl or rebooting.

| Posted on 2016-09-30 at 15:34:11 UTC by Andrew Ayer | Reply to This

I'm a systemd maintainer and have published some of my thoughts in a response: https://www.facebook.com/notes/david-timothy-strauss/how-to-throw-a-tantrum-in-one-blog-post/10108242959884190

This isn't an official response from the project, just one from someone who is very familiar with the issues in question.

| Posted on 2016-09-30 at 08:02:32 UTC by Reader David Strauss | Reply to This

At the request of a few readers, I've moved the post to a more sharable/annotation-friendly form on Medium:

https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post-c2ccaa58661d

| Posted on 2016-09-30 at 11:07:52 UTC by Reader David Strauss | Reply to This

Dear David, you wrote so many words, but the essence of your article is simple "You are wrong! Why? Because!".

| Posted on 2016-09-30 at 14:19:59 UTC by Reader Andrey Bergman | Reply to This

With a serving of ad hominem fallacies, strawmen, and dishonest arguments! Is this the best rebuttal systemd can muster?

| Posted on 2016-10-01 at 16:16:14 UTC by Andrew Ayer | Reply to This

Is making crass statements like "defective by design" all you have? if you hate systemd so much, don't use it. By your measure every program that has a bug is "defective by design".

| Posted on 2016-10-01 at 17:30:31 UTC by Reader trivia | Reply to This

Is making crass statements like "defective by design" all you have?

Nah, it's not like I wrote a blog post explaining my position with evidence and logical arguments and stuff.

if you hate systemd so much, don't use it.

I don't, and I hope others don't also.

By your measure every program that has a bug is "defective by design".

Try harder.

| Posted on 2016-10-01 at 19:42:13 UTC by Andrew Ayer | Reply to This

This bug, which I was not able to reproduce on Kali with 215, is hardly related to the other shortcomings for which you despise systemd. The only possibly related one is 'complexity' ... so complex someone was forced to make a < vs <= mistake? plz

| Posted on 2016-10-04 at 01:19:36 UTC by Anonymous | Reply to This

Kali appears to be unstable now: unable to login via UI

| Posted on 2016-10-04 at 01:48:13 UTC by Anonymous | Reply to This

That is why I don't see any reason for discussing article by David Strauss. He, just like most systemd supporters lies from the very beginning.

| Posted on 2016-10-01 at 20:56:52 UTC by Reader Andrey Bergman | Reply to This

This is a fair point, but it also isn’t a fundamental design element worthy of calls to abandon systemd. As most, it would justify a call to fork systemd and reverse the umask default.

I liked this bit. Because it is a fundamental design element when the developer team developing your software are completely unwilling to consider any suggestions from the outside. And you know that, because you suggested a fork right away, which of course, never would gain any traction if the fork is only about default policies.

I bet If I were to open a bug proposing to reverse the default behavior to a fail-safe behavior it would just be closed as wontfix with one or more arrogant "systemd is perfect as it is" comments below it. This gives good indications towards the design goals of systemd. It is not meant to be a solid and reliable foundation for Linux systems. That would be boring. It is meant to be an attractive sham-bandwagon to jump onto, selling the feel of flashy and new. Fail-safe behavior gets in the way of that, even if systemd cannot uphold the things it promises for every user.

So yes, it is worth abandoning systemd exactly due to issues like this.

| Posted on 2016-10-02 at 02:59:30 UTC by Anonymous | Reply to This

What a brilliant rebuttal - I read as far as "minor security issue".

This exactly validates the critisism of the systemd-mindset: Any non-privileged process being trivially able to de-facto crash a system is a "minor security issue".

Case closed.

| Posted on 2016-10-02 at 07:59:30 UTC by Reader Eugen Rieck | Reply to This

I stopped reading the message from David Strauss at

“Ayer’s work shows lots of promise — he’s clearly a talented security and software engineer — but…”

That’s deep in personal attacks and rhetorics.

| Posted on 2016-10-06 at 09:03:18 UTC by Anonymous | Reply to This

Did you address why this bug has existed for 2 years? Have you addressed magic numbers and why they appear?

| Posted on 2016-10-02 at 06:50:19 UTC by Anonymous | Reply to This

Instead of make it complicated with systemd, make it uncomplicated with https://en.wikipedia.org/wiki/OpenRC

| Posted on 2016-09-30 at 09:01:25 UTC by Anonymous | Reply to This

Reading the article makes me wonder whether the nsa is actively paying Lennart. As the situation we're now in is so obviously an american trait, something enormous and so reputedly awesome and terrible only depending on who you ask.

| Posted on 2016-09-30 at 10:36:17 UTC by Reader mar77i | Reply to This

I'm an American, I don't code like that. I'm also a sysadmin by trade going on 25 years now. Does that mean I am or am not classified by your 'American traits'?

| Posted on 2016-10-02 at 00:00:47 UTC by Anonymous | Reply to This

Please try not to be offended. He merely is referring to deep state/CIA/NSA -whatever mashup.

Then there are the normal people of USA. The world likes the regular usa citizen as a rule of thumb.

| Posted on 2016-10-04 at 18:52:06 UTC by Reader vilsu33 | Reply to This

Thank you, and the commenters, for providing tangible support to my instinctual distaste of systemd. Maybe it will encourage people to take another look at the BSD's.

| Posted on 2016-09-30 at 19:23:09 UTC by Reader cvvs | Reply to This

Hey folks... try http://smarden.org/runit/ as init-scheme

| Posted on 2016-10-01 at 08:43:15 UTC by Anonymous | Reply to This

This does not work. Tested with many systemd versions and distributions.

# Failed to notify init system: Connection refused

This article and most comments are only FUD.

Go and shoot yourself stupid haters

| Posted on 2016-10-01 at 12:05:59 UTC by Anonymous | Reply to This

That message means systemd has already crashed, but you're probably too much of a flaming nutjob to understand that.

| Posted on 2016-10-01 at 16:16:37 UTC by Andrew Ayer | Reply to This

how to make fool of yourself in one tweet https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post-c2ccaa58661d

| Posted on 2016-10-01 at 14:17:24 UTC by Anonymous | Reply to This

What systemd supporters lack in logical skills, they make up for by repeating themselves.

| Posted on 2016-10-01 at 16:17:39 UTC by Andrew Ayer | Reply to This

Or repeating Poettering's The Correct way of how world is supposed to be.

| Posted on 2016-11-06 at 05:09:06 UTC by Anonymous | Reply to This

Andrew, thank you for this very lucid article.

I have been philosophically uneasy with systemd ever since it rolled out on Arch and have read many opinions and counter-opinions on the matter. A few months ago I put a Manjaro spin (Fluxbox & openRC) on my laptop and everything worked just fine. I think, in light of the above arguments, I've reached the tipping point to take a deep breath and point my primary Arch system to the openRC alternatives. Thank you.

| Posted on 2016-10-01 at 14:30:36 UTC by Reader Matt Shulver | Reply to This

What most people do not know is that it's possible to purge systemd from Ubuntu 16.06 and everything works well (I actually had to do this because the wiser systemd would be waiting for ethernet network to come up, even with a disconnected cable).

systemd is a bad implementation of what had already been achieved successfuly with SMF https://en.wikipedia.org/wiki/Service_Management_Facility

| Posted on 2016-10-02 at 00:11:11 UTC by Reader estan | Reply to This

How long before Pitt and crew gets what they want and purge that option under "too much hassle to maintain"?

| Posted on 2016-10-09 at 13:53:14 UTC by Anonymous | Reply to This

Hmm, you mean software has bugs? Why would systemd be any different. Now that the bug has been pointed out, I'm sure a fix will be available RSN... so thanks for helping make systemd better!

| Posted on 2016-10-02 at 01:07:56 UTC by Reader Scott Dowdle | Reply to This

Software generally has N-bugs-per-X-lines-of-code, with N and X varying by program type and the care of the development team.

However a program with 20x the lines of code as another program almost always has more bugs. Systemd has about 20x the lines of code as OpenRC.

| Posted on 2016-10-06 at 09:09:21 UTC by Anonymous | Reply to This

A nice side effect from all the "bruhaha" of this hack is you can see who works for NSA and who doesn't.

| Posted on 2016-10-02 at 01:28:37 UTC by Anonymous | Reply to This

Totally agree with you Andrew. Nicely summed up.

Recently i kicked Arch Linux to the curb and switched to Void Linux because of systemd.

Regards

| Posted on 2016-10-02 at 07:24:38 UTC by Reader iskander | Reply to This

This is typical for systemd developers - force it onto everyone, then refuse to fix fatal bugs.

You can see this in countless issue reports at their tracker.

The only sane conclusion is to avoid systemd.

| Posted on 2016-10-02 at 08:38:11 UTC by Reader mark | Reply to This

So then please give some concrete examples in the systemd bug tracker!

| Posted on 2016-10-02 at 08:48:16 UTC by Reader Stefan | Reply to This

nobody asked for systemd. go and fix it yourself.

| Posted on 2016-10-02 at 13:00:55 UTC by Anonymous | Reply to This

the command does not crash centos 7- 1511. checked.

| Posted on 2016-10-02 at 10:12:25 UTC by Reader idan | Reply to This

try this on fedora25. I don'y see any crash

| Posted on 2016-12-01 at 18:55:37 UTC by Reader vedeheme | Reply to This

'while true ; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify "" ; done' do crash centos 7- 1511. checked.

| Posted on 2016-10-02 at 11:49:39 UTC by Anonymous | Reply to This

If its needed it will result in a reinstall

I just removed the systemd And it drags the whole desktop environment with it.. into zerobe

will post the results if there is anything to note

be gone... daemon!

| Posted on 2016-10-02 at 15:55:14 UTC by Reader herauthon | Reply to This

As a security analyst for a Fortune 50 company, you can bet I'll be trying this formally. A rogue employee DoSing a system? That's a big problem for us. We throw a lot of money at Red Hat; a hole like this would set off a lot of panic for our IT teams, and some of that will be aimed at RH. Getting an unprivileged shell prompt isn't all that hard, sadly. Parlaying that into a DoS with a short one-liner is a HUGE issue. Strauss' reply isn't reassuring, it's frightening... and will likely make its way into my gap writeup, so the CISO can consider it. Our business is all about managing risk, and it sounds like Red Hat's systemd team is taking, and exposing us to, unnecessary risk.

| Posted on 2016-10-02 at 17:31:43 UTC by Anonymous | Reply to This

Sorry I am new to the systemd wars and there is something that I do not understand: If most people like the features that systemd offers but many do not like how they are being implemented, why there is not a fork?

| Posted on 2016-10-02 at 17:57:04 UTC by Anonymous | Reply to This

The issues people have with systemd mean forking it wouldn't solve the problem. Systemd flies in the face of Unix philosophy, by having one program try to do it all. Forking it would never fix that, and there are other good alternatives out there such as runit and openrc. There was a fork at one point called uselessd, where the developers tried to remove as much of the unnecessary code in systemd as possible. The problem was that systemd was so large by then that maintaining the fork with only a few people was too much work, so it died. If you don't like systemd, the best option is to just not use it. There are other programs that can do the same thing as systemd, it's just you have to use more than one program (which is the smarter way to do it imo, with systemd you have a single point of failure, which is not good, and with as much code as systemd has, more errors are more likely to occur).

| Posted on 2016-10-02 at 18:18:05 UTC by Anonymous | Reply to This

Most people don't care as long as it works.

Of those that care most people both like what it offers an are relatively happy with it's implementation.

No doubt systemd is a major change from sysvinit and some people just don't like change.

| Posted on 2016-10-03 at 10:52:26 UTC by Anonymous | Reply to This

People hate stupid and bad change, and for good reason.

Systemd is an abortion. Poettering is a either a short-sighted fool or a malicious tool.

| Posted on 2016-10-03 at 18:00:55 UTC by Reader AC | Reply to This

I think the word you meant to use is 'abomination'.

| Posted on 2016-10-04 at 02:55:09 UTC by Anonymous | Reply to This

most people just want their programs to work. Systemd was pushed as a dependency of Gnome - which many people need - at least partly due to systemd devs having a say in Gnome development.

| Posted on 2016-10-06 at 09:12:15 UTC by Anonymous | Reply to This

Because:

A: there are already multiple alternatives if all you want it dependency based init. But the scope of systemd is creeping ever wider.

B: thanks to the scope creep more and more depends on systemd being present (particularly the stuff coming out of Freedesktop). Gnome for instance wants logind to be present to handle "sessions". These days Logind is responsible for handling sleep and hibernate, with formerly independent powerd/powerkit acting as a compatibility shim. Wayland will depend on Logind and Polkit for managing access to tings like keyboard and GPU.

| Posted on 2016-10-09 at 13:59:23 UTC by Anonymous | Reply to This

The magic number thing reminds me of sony's little gem. . .

int getRandomNumber() { return 4; }

| Posted on 2016-10-02 at 22:13:16 UTC by Reader Justme | Reply to This

yeah i just mindlessly opened a terminal and typed that exact command after breakfast while i was still only partially awake. thanks for the warning. i also find it is not good if you highlight all the folders in the / directory then delete them late at night. havent tried this after breakfast

| Posted on 2016-10-03 at 03:32:57 UTC by Reader mary | Reply to This

The difference is that this command can be performed as any user. To delete everything in /, you need to be root.

| Posted on 2016-10-12 at 03:25:41 UTC by Reader Danielle McLean | Reply to This

Fedora released fixed systemd package for Fedora 24 today. Expect other updates RSN if they haven't already been released. Look, a bug in software that got fixed. Expect billions more in every piece of software ever written.

| Posted on 2016-10-03 at 04:19:20 UTC by Reader Scott Dowdle | Reply to This

Systemd is a horrible thing for Linux. All you ubuntu n00bs have caused this

| Posted on 2016-10-03 at 16:32:34 UTC by Reader yo dawg | Reply to This

Actually Ubuntu was a late adopter. The 1337 distros Fedora and Arch were the early adopters...

| Posted on 2016-10-09 at 14:01:06 UTC by Anonymous | Reply to This

Been saying it for a while...systemd is ready made for a rootkit. And it will be the mother of all rootkits when it happens.

But no matter, the systemd astroturf brigade will convince us to "go to freebsd" if we don't like it, right?

| Posted on 2016-10-03 at 20:23:58 UTC by Reader chickenhead | Reply to This

Could have sworn there is already a systemd-dependent bitcoin worm floating around...

| Posted on 2016-10-09 at 14:02:01 UTC by Anonymous | Reply to This

nothing happened after running the command in ubuntu 16.04

| Posted on 2016-10-05 at 21:08:34 UTC by Anonymous | Reply to This

You forgot that the most important reason for not using it is that it is totally undebuggable. You cannot strace it, cannot gdb it etc... Just because it's PID1. Can you imagine a system without strace ? It exists with systemd cancering more and more of your system, and it's a real pain each time I end up on such a self-sustaining system which refuses to cooperate with humans.

| Posted on 2016-10-06 at 21:44:21 UTC by Reader Willy | Reply to This

WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO.

| Posted on 2016-10-08 at 03:26:57 UTC by Anonymous | Reply to This

Have fun debugging with those binaries - Oh they are SO much better than scripts. Too bad the millennials destroyed linux with systemd.

| Posted on 2016-10-08 at 03:28:37 UTC by Anonymous | Reply to This

As well as Devuan other NON Systemd alternatives are:

Manjaro OpenRC (Arch Linux desktop) https://sourceforge.net/projects/manjaro-openrc/

Alpine Linux (also uses openrc) http://alpinelinux.org/

| Posted on 2016-10-08 at 15:10:26 UTC by Reader Stuart | Reply to This

Linux guru talks about the Linux desktop https://www.youtube.com/watch?v=MpDdGOKZ3dg

| Posted on 2016-10-15 at 08:34:06 UTC by Reader Anonymus | Reply to This

you know i have said this a long time .at anyrate the funny thing i have been able to take down any linux server or windows server via one simple process..or rather 2 ..telling them to do 2 actual things/processes at the exact same time in real time..not so u say..haha ask Godaddy Ask Host Monster also. Oh life is Good when own an Amiga using a simple program called termite..now did i do it ..on purpose no .The point is it can be done ..This proves my point all along x86 system cant multitask in true real-time.This problem is proving it.

| Posted on 2016-10-17 at 02:33:10 UTC by Reader truthsayer | Reply to This

This made my day. I am stuck with CentOS 6 because I'll never countenance systemd (or pulseaudio). Thank you Ayer!!!

| Posted on 2016-10-21 at 02:30:20 UTC by Reader Strichard Hallman | Reply to This

seem this issue has been fixed, run while true; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""; done

for 30 minutes on FC24 4.5.5-300.fc24.x86_64 on virtual box with 1.5GB Ram still running fine. Load Average 1.48

| Posted on 2016-11-01 at 16:36:55 UTC by Anonymous | Reply to This

Ok, above was too hasty conclusions, ssh is fine, but VM console was unresponsive, also when try to reboot Vm its say

Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out

Failed to open /dev/initctl: No such device or address Failed to talk to init daemon.

daang, issue still not fixed.

| Posted on 2016-11-01 at 16:49:41 UTC by Anonymous | Reply to This

RHEL 7.3 is live today but bug still not fixed.

| Posted on 2016-11-03 at 09:59:14 UTC by Anonymous | Reply to This

how's this for a stake in the ground: I will never hire anyone who has contributed to systemd. it's only me in my capacity as a software architect with hiring authority, but if you have a commit in the systemd mess, you got a black ball. I will check.

| Posted on 2016-12-09 at 02:51:03 UTC by Anonymous | Reply to This

Post a Comment

Your comment will be public. If you would like to contact me privately, please email me. Please keep your comment on-topic, polite, and comprehensible. Use the "Preview" button to make sure your comment is properly formatted. Name and email address are optional. If you specify an email address it will be kept confidential.

Post Comment


(Optional; will be published)


(Optional; will not be published)


(Optional; will be published)


  • Blank lines separate paragraphs.
  • Lines starting with ">" are indented as block quotes.
  • Lines starting with two spaces are reproduced verbatim.
  • Text surrounded by *asterisks* is italicized.
  • Text surrounded by `back ticks` is monospaced.
  • URLs are turned into links.