Skip to Content [alt-c]

Andrew Ayer


Systemd is not Magic Security Dust

Systemd maintainer David Strauss has published a response to my blog post about systemd. The first part of his post is replete with ad hominem fallacies, strawmen, and factual errors. Ironically, in the same breath that he attacks me for not understanding the issues around threads and umasks, he betrays an ignorance of how the very project which he works on uses threads and umasks. This doesn't deserve a response beyond what I've called out on Twitter.

In the second part of his blog post, Strauss argues that systemd improves security by making it easy to apply hardening techniques to the network services which he calls the "keepers of data attackers want." According to Strauss, I'm "fighting one of the most powerful tools we have to harden the front lines against the real attacks we see every day." Although systemd does make it easy to restrict the privileges of services, Strauss vastly overstates the value of these features.

The best systemd can offer is whole application sandboxing. You can start a daemon as a non-root user, in a restricted filesystem namespace, with mandatory access control. Sandboxing an entire application is an effective way to run potentially malicious code, since it protects other applications from the malicious one. This makes sandboxing useful on smartphones, which need to run many different untrustworthy, single-user applications. However, since sandboxing a whole application cannot protect one part of the application from a compromise of a different part, it is ineffective at securing benign-but-insecure software, which is the problem faced on servers. Server applications need to service requests from many different users. If one user is malicious and exploits a vulnerability in the application, whole application sandboxing doesn't protect the other users of the service.

For concrete examples, let's consider Apache and Samba, two daemons which Strauss says would benefit from systemd's features.

First Apache. You can start Apache as a non-root user provided someone else binds to ports 443 and 80. You can further sandbox it by preventing it from accessing parts of the filesystem it doesn't need to access. However, no matter how much you try to sandbox Apache, a typical setup is going to need a broad amount of access to do its job, including read permission to your entire website (including password-protected parts) and access to any credential (database password, API key, etc.) used by your CGI, PHP, or similar webapps.

Even under systemd's most restrictive sandboxing, an attacker who gains remote code execution in Apache would be able to read your entire website, alter responses to your visitors, steal your HTTPS private keys, and gain access to your database and any API consumed by your webapps. For most people, this would be the worst possible compromise, and systemd can do nothing to stop it. Systemd's sandboxing would prevent the attacker from gaining access to the rest of your system (absent a vulnerability in the kernel or systemd), but in today's world of single-purpose VMs and containers, that protection is increasingly irrelevant. The attacker probably only wants your database anyways.

To provide a meaningful improvement to security without rewriting in a memory-safe language, Apache would need to implement proper privilege separation. Privilege separation means using multiple processes internally, each running with different privileges and responsible for different tasks, so that a compromise while performing one task can't lead to the compromise of the rest of the application. For instance, the process that accepts HTTP connections could pass the request to a sandboxed process for parsing, and then pass the parsed request along to yet another process which is responsible for serving files and executing webapps. Privilege separation has been used effectively by OpenSSH, Postfix, qmail, Dovecot, and over a dozen daemons in OpenBSD. (Plus a couple of my own: titus and rdiscd.) However, privilege separation requires careful design to determine where to draw the privilege boundaries and how to interface between them. It's not something which an external tool such as systemd can provide. (Note: Apache already implements privilege separation that allows it to process requests as a non-root user, but it is too coarse-grained to stop the attacks described here.)

Next Samba, which is a curious choice of example by Strauss. Having configured Samba and professionally administered Windows networks, I know that Samba cannot run without full root privilege. The reason why Samba needs privilege is not because it binds to privileged ports, but because, as a file server, it needs the ability to assume the identity of any user so it can read and write that user's files. One could imagine a different design of Samba in which all files are owned by the same unprivileged user, and Samba maintains a database to track the real ownership of each file. This would allow Samba to run without privilege, but it wouldn't necessarily be more secure than the current design, since it would mean that a post-authentication vulnerability would yield access to everyone's files, not just those of the authenticated user. (Note: I'm not sure if Samba is able to contain a post-authentication vulnerability, but it theoretically could. It absolutely could not if it ran as a single user under systemd's sandboxing.)

Other daemons are similar. A mail server needs access to all users' mailboxes. If the mail server is written in C, and doesn't use privilege separation, sandboxing it with systemd won't stop an attacker with remote code execution from reading every user's mailbox. I could continue with other daemons, but I think I've made my point: systemd is not magic pixie dust that can be sprinkled on insecure server applications to make them secure. For protecting the "data attackers want," systemd is far from a "powerful" tool. I wouldn't be opposed to using a library or standalone tool to sandbox daemons as a last line of defense, but the amount of security it provides is not worth the baggage of running systemd as PID 1.

Achieving meaningful improvement in software security won't be as easy as adding a few lines to a systemd config file. It will require new approaches, new tools, new languages. Jon Evans sums it up eloquently:

... as an industry, let's at least set a trajectory. Let's move towards writing system code in better languages, first of all -- this should improve security and speed. Let's move towards formal specifications and verification of mission-critical code.

Systemd is not part of this trajectory. Systemd is more of the same old, same old, but with vastly more code and complexity, an illusion of security features, and, most troubling, lock-in. (Strauss dismisses my lock-in concerns by dishonestly claiming that applications aren't encouraged to use their non-standard DBUS API for DNS resolution. Systemd's own documentation says "Usage of this API is generally recommended to clients." And while systemd doesn't preclude alternative implementations, systemd's specifications are not developed through a vendor-neutral process like the IETF, so there is no guarantee that other implementers would have an equal seat at the table.) I have faith that the Linux ecosystem can correct its trajectory. Let's start now, and stop following systemd down the primrose path.

Hi, I'm Andrew. I'm the founder of SSLMate, a service which automates your SSL certificate deployment. I also develop open source projects like git-crypt and titus.

I blog here about a variety of technology topics, including security, devops, IPv6, and reliable programming. If you liked this post, check out my other posts or subscribe to my Atom feed.

My email address is I'm AGWA at GitHub and @__agwa on Twitter.


The comments below are owned by whoever posted them. I am not responsible for them in any way.

More: Even assuming sandboxing is a panacea ( it isn't, but it's a good thing anyway ), the assumption that systemd make it easy and without systemd it isn't easy it's false.

There are plenty of easy to use tools, let me say for example firejail but many others works too, that implement sandboxing in easy and accessible way, and probably thanks to a better awareness, even better than what is usually done with systemd.

| Posted on 2016-10-02 at 21:28:44 UTC by Reader nextime | Reply to This

This (and your previous post) has to be one of the best and most well thought-out posts about why we should not be using systemd. Most people seem to hate systemd for religious reasons but these are excellent scientific reasons. Bravo!

| Posted on 2016-10-02 at 22:30:40 UTC by Reader JD | Reply to This

Here you go:

| Posted on 2016-10-02 at 22:55:46 UTC by Reader David Strauss | Reply to This

Why do this read like a mailing list argument in blog form?

| Posted on 2016-10-03 at 08:41:34 UTC by Anonymous | Reply to This

Damn, Ayer got wrecked by David's response

Kid doesn't have a clue.

| Posted on 2016-10-03 at 08:17:54 UTC by Anonymous | Reply to This

It is more like Strauss doesn't have a clue at all. He resorts to ad-hominem again (like you just did with the "kid" attack), picks sentences out of context, answers things that weren't even asked, does a typical systemd apologist post - when in doubt, non-sequitur, ad-hominem, fud.

| Posted on 2016-10-03 at 12:50:46 UTC by Anonymous | Reply to This

He also makes mention that "OpenSSH on Fedora ships with systemd" and infers that this is OK. Wait, you work for Red Hat, they pay you and also ship Fedora? Bring me back to where this matters???

| Posted on 2016-10-04 at 12:07:16 UTC by Anonymous | Reply to This


| Posted on 2016-10-03 at 15:18:51 UTC by Andrew Ayer | Reply to This

He, he. Strauss simply doesn't understand, or doesn't want to see, that sandboxing is one thing. However, privilege separation and design in an application is another. The former does not protect an application from getting completely owned.

The fact that you've got a systemd developer talking like this is exceptionally worrying. He also seems to think a non-privileged user being able to do a denial of service on the system is a 'trivial security problem'. Is it me or do systemd's developers not get the concept of a multi-user system or anything Unix systems have had for decades?

| Posted on 2016-10-04 at 14:01:00 UTC by Reader DV | Reply to This

He is likely inspired by the latest out of academia, unikernels.

Effectively DOS in a VM...

| Posted on 2016-10-05 at 12:14:30 UTC by Anonymous | Reply to This

I am not well-versed in security matters, but I want to make my best. So I read both your post and the latest answer of Strauss.

The result is that I understand your position and arguments, while the the post of Strauss leaves me utterly confused.

Which can only lead me to conclude that you are right, since clarity of design seems to be the top quality in terms of security.

So thanks for the warning.

| Posted on 2016-10-03 at 14:42:56 UTC by Anonymous | Reply to This

Glad to see you mention qmail, and his response to your comment was fairly hilarious. Privilege separation via exec chaining (aka Berstein Chaining) is a key concept that the systemd developers don't get. If they DID get it, they'd understand that that's precisely why you don't run all of this in the same process. Small, simple utilities that do what they need to do, drop privileges, and execute something else.

Supervise > tcpserver > drop privs > app authentication > remainder of the app. This is the antithesis of the systemd model. If they truly understood it, they'd understand the importance of a tiny PID 1, which serves as the basis for everything else: small, secure tools which do precisely and only what they need to do, remove their privileges, and chain to the next step.

| Posted on 2016-10-03 at 18:37:44 UTC by Reader JCC1 | Reply to This

There's a couple of possibilities. Either they don't understand this, which I don't think they do anyway, or they deliberately don't want to do this.

If they were to turn PID 1 into something simple that basically spawned processes then there is the possibility that a replacement could be made, replacements for components could be made or alternative init systems can keep getting used. By controlling PID 1 from the kernel all the way down and software then starting to rely on systemd you are hard pushed to get this thing dropped or replaced.

If PID 1 is the way it is for political reasons then it's the worst possible reason, because they have completely sacrificed any sort of priorities on security for politics. Once it becomes more widespread in various Linux systems, embedded systems and network devices, this is going to blow up in all our faces.

| Posted on 2016-10-04 at 15:13:22 UTC by Reader DV | Reply to This

Andrew says systemd has got to where it is by marketing, but I don't think that's true. It's politics, and one of the most political piece of software we've ever seen in the open source world, and that's saying something.

| Posted on 2016-10-04 at 15:30:43 UTC by Reader DV | Reply to This

Marketing and politics overlap to a increasing degree.

Frankly what is going on is that systemd devs envision only two use cases, desktop and containers (the latter being a way for them to gain some appearance of credibility). Anything else is either archaic or "wrong".

And all of this is wrapped in a massive amount of paternalism that originates with Gnome and has spread across the larger Linux userspace via Freedesktop.

On top of this, the core devs of systemd is backed by Red Hat's "deep" pockets. And something shifted within RH some years ago and now they seem to have a unstated goal of controlling the Linux ecosystem by generating "code churn".

Meaning that they get to say the direction of the ecosystem with tactic not that dissimilar from Microsoft's Embrace Extend Extinguish.

They have key personnel embedded within various projects, and keep introducing various changes between releases that they coordinate via company channels. Thus third parties find themselves either chasing the RH dust, or turn the keys over to RH.

| Posted on 2016-10-06 at 11:27:49 UTC by Anonymous | Reply to This

This Jeff Johnson bug that led to a RPM fork is probably worth mentioning:

| Posted on 2017-01-25 at 04:33:11 UTC by Reader Yuhong Bao | Reply to This

It is puzzling that Mr Strauss doesn't appreciate the points you are making - the argument goes back many years. Maybe a picture might help ? The RAID conference proceedings of 2001 has one on page 38...

| Posted on 2016-10-03 at 21:46:45 UTC by Reader Anonymous | Reply to This

Save for Slackware, every mainstream and/or popular distribution accepted systemd. Linus Torvalds himself is only able to - or willing to - grumble about the binary logs and ignores both the users' screams and the rest of the catastrophes systemd creates.

For now, save for RedHat stuff (RHEL, CentOS, Fedora) we can get rid of systemd manually, but for how long? What if the softwares will begin to require systemd? Because of stupidity, or because if they don't support systemd, then their software won't run properly or won't run at all. GNOME3, one of the most used desktop environments already depends on systemd and the trend will continue. (And to make it worse, the package maintainers link systemd as dependency even for programs/libraries actually not depending on systemd, but by this they prevent you to install the package without installing systemd.) Like with PulseAudio. Tons of Linux software depends on that unnecessary and bloated layer over OSS/ALSA. And when most of the Linux software will require systemd, then the entire Linux ecosystem will depend on systemd and we will not have the choice to ditch systemd, but to accept it and let it taking over our systems or moving to some BSD or Solaris (or whatever you can find).

Whatever Poettering creates is fundamentally broken and yet, whatever he creates, will be accepted as a quasi-standard in the Linux world. I do not want to create CT-s here, i am merely stating some facts. But i would not be surprised, when the events i mentioned above shall happen, then RH will close the sources of systemd. And the users will have the illusion, that they have an open source OS and they running open source programs, but in reality, they have a blackbox between the two, what can do anything, what it's creators want.

It seems to me, that whoever who try to oppose systemd is waging a losing war. Linux is lost. It's much less painful, to slowly switch to another UNIX step by step now, than do it instantly when systemd ultimately conquered Linux.

Sorry for pouring this much bullshit here.

| Posted on 2016-11-10 at 08:57:52 UTC by Reader Wrzi | Reply to This

Free/OpenBSD is a great alternative and still blessedly systemd-free.

| Posted on 2016-11-27 at 10:50:01 UTC by Anonymous | Reply to This

To provide a meaningful improvement to security without rewriting in a memory-safe language,

If you sincerely believe that rewriting Apache or any other piece of software in a memory-safe language would make it inherently more secure, I'll have to agree with David's assessment of your inexperience.

| Posted on 2016-11-27 at 17:09:12 UTC by Reader DES | Reply to This

If you sincerely believe that rewriting [...] software in a memory-safe language would make it inherently more secure [...]

(emphasis mine)

Ooh, that's a nice strawman you've got there.

The author wrote:

To provide a meaningful improvement to security without rewriting in a memory-safe language, Apache would need to [...]

Which doesn't state that rewriting is the only thing needed to make it more secure. You're misrepresenting his "rewriting is one possible necessary condition, but I do not state whether it is a sufficient one" to "rewriting is one possible necessary and sufficient condition to making the software safer".

Not sure whether to attribute this to malice or accidental (or emotion-influenced) lack of reading comprehension.

Given the history of tactics employed by pro-systemd (and to be fair, anti-systemd) folks, it's getting harder to give the benefit of doubt. But I (and others, I hope) should strive to keep the discussion free of fame wars and intellectual dishonesty, right?

| Posted on 2016-12-28 at 09:22:46 UTC by Anonymous | Reply to This

Only another ignorant systemd hater.

Pro tip: More valid arguments, less hate.

| Posted on 2017-03-02 at 07:16:15 UTC by Reader Nick | Reply to This

Yup, look at this firefox 52 requires libpulse (pulseaudio) that requires systemd. F4k this shit.

| Posted on 2017-03-08 at 00:34:20 UTC by Anonymous | Reply to This

systemd sucks, only supported by company. microsoft-way not going to work.

| Posted on 2017-03-15 at 21:57:02 UTC by Anonymous | Reply to This

Uhm... about Apache:

1. "Broad" access? No. A site exists under a directory. Of course the httpd will have access to files it is, uhm... serving.

2. Credentials used by PHP? Nope. Only if you run PHP DSO which you shouldn't. The web server can have zero access to PHP files, because properly configured fastcgi daemon (eg. php-fpm) will get a fastcgi REQUEST sent to by httpd. It is then the PHP itself that finds and opens a file for reading, parsing and execution.

3. Attacker gaining remote execution of httpd will be able to do all those things regardless of an init system. HOWEVER, systemd ALLOWS very simple and easy, all within the one unit file, configuration of seccomp filters, capability dropping, binding to <1000 ports for daemons not running as root, etc... I don't think any other init system does this? Oh, sure, it's not the domain of an init system to do all this? Think again, it exactly is, because init is managing processes.

It's hard to take the rest of your post seriously when you make such huge mistakes.

| Posted on 2017-06-03 at 14:36:32 UTC by Reader infosec | Reply to This

Post a Comment

Your comment will be public. If you would like to contact me privately, please email me. Please keep your comment on-topic, polite, and comprehensible. Use the "Preview" button to make sure your comment is properly formatted. Name and email address are optional. If you specify an email address it will be kept confidential.

Post Comment

(Optional; will be published)

(Optional; will not be published)

(Optional; will be published)

  • Blank lines separate paragraphs.
  • Lines starting with ">" are indented as block quotes.
  • Lines starting with two spaces are reproduced verbatim.
  • Text surrounded by *asterisks* is italicized.
  • Text surrounded by `back ticks` is monospaced.
  • URLs are turned into links.