Our entire team will be at FOSDEM, and we'd be thrilled to meet more of the Mullvad team. Protecting systems like yours is core to us. We want to understand how we put the right roots of trust and observability into your hands.
Edit: I've reached out privately by email for next steps, as you requested.
Hi David. Great! I actually wasn't planning on going due to other things, but this is worth re-arranging my schedule a bit. See you later this week. Please email me your contact details.
As I mentioned above, we've followed systemd's development in recent years with great interest, as well as that of some other projects. When I started(*) the System Transparency project it was very much a research project.
Today, almost seven years later, I think there's a great opportunity for us to reduce our maintenance burden by re-architecting on top of systemd, and some other things. That way we can focus on other things. There's still a lot of work to do on standardizing transparency building blocks, the witness ecosystem(**), and building an authentication mechanism for system transparency that weaves it all together.
I'm more than happy to share my notes with you. Best case you build exactly what we want. Then we don't have to do it. :)
> because upstart could grok the scripts without change
I don't believe that's the case. The "service" command (which is part of the sysvinit-utils package, not Upstart) invokes either Upstart or SysV init as necessary, but Upstart itself has no awareness of the SysV init world. You couldn't have an Upstart service depend on a SysV init one, list SysV service status with Upstart, or enable a SysV service through Upstart.
In case you're curious, the wrapping on the systemd side is more comprehensive. SysV scripts appear as units, and systemd parsed the semi-standard metadata at the top of most SysV init files to determine when the service should start if enabled (translating from the traditional run levels). As units, the SysV init scripts are possible to enable/disable, start/stop, and list using the standard systemd commands. They can also participate in dependency graphs alongside systemd units.
> it's not exactly a good choice for embedded systems.
I don't develop embedded systems, but systemd is actually popular in that space because of its watchdog capabilities and its inclusion in projects like GenIVI and Tizen.
I see nothing there that can't be achieved by a simple supervision suite. s6, nosh, even runit provide those features; the extra complexity of systemd isn't needed at all.
If even embedded developers are getting pulled in by the sirens of systemd because OH MY GOSH IT HAS WATCHDOG CAPABILITIES, then we definitely need to work more on raising awareness about supervision.
Socket activation doesn't have any systemd-based interface. You just get a file descriptor passed in the normal Unix way. The systemd library functions related to socket activation are utility functions for examining the inherited socket, but they are just wrappers for any other way you might do so.
You can configure daemons like nginx or PHP-FPM to use sockets inherited from systemd instead of their own, and it works fine. They don't have any specific support for systemd socket activation, nor do they need to. They can't even tell the difference between the systemd sockets and ones they'd get on a configuration reload.
The closest I could find in the docs to what digi_owl said is the following:
> Internally, these functions send a single datagram with the state string as payload to the AF_UNIX socket referenced in the $NOTIFY_SOCKET environment variable. If the first character of $NOTIFY_SOCKET is "@", the string is understood as Linux abstract namespace socket. The datagram is accompanied by the process credentials of the sending service, using SCM_CREDENTIALS.
I can see how someone would be reluctant to rely on that, even given the interface promise and the nudging of the systemd developers. To be more consistent with what's a stable, public interface and the admonition to avoid internals, I would probably drop the word "internally."
However, even with your change, I still read that section as describing implementation detail that's not guaranteed to be stable. If that note describes a stable, documented protocol, a link back to the documentation of that protocol would be helpful and reassuring.
I don't think Red Hat was the driving force behind systemd's container and VM integration.
systemd-nspawn was created to provide rapid testing of systemd, and it was (and, for now, still is) marked as an experimental utility that isn't for production purposes.
If you look at many of the early blog posts mentioning systemd and containers/VMs, you'll actually see my work and my company (Pantheon) mentioned more than Red Hat. In more recent posts, you'll see lots of CoreOS influence. Project Atomic isn't working with systemd in any direct, significant way; you won't see much coordination other than in some shared philosophy about a stateless base OS, and even that is more attributable to CoreOS's influence than Red Hat's.
I think the influence of Red Hat on systemd is overstated. Of the two perspectives: (1) Red Hat driving everything and (2) Lennart answering mostly to himself, the latter is way more accurate, for better or worse (depending on your opinion of him).
Well, Red Hat were quick to adopt CoreOS' appc, before a lot of companies ultimately united under the Open Container Project banner.
As for:
and it was (and, for now, still is) marked as an experimental utility that isn't for production purposes.
rkt used nspawn for a while, and it's still systemd-based.
The fact that Lennart was giving out presentations about nspawn specifically makes me believe it's very much intended to be used in production, as a "chroot on steroids".
> The fact that Lennart was giving out presentations about nspawn specifically makes me believe it's very much intended to be used in production, as a "chroot on steroids".
This is a recent development -- and why I put the caveat "for now." systemd-nspawn is probably going to be marked production-ready quite soon, especially because it's the foundation for the CoreOS Rocket container tool.
If that were the case, wouldn't the author support the kdbus work? It seems like they're not a fan of that, either, given the (misleading) complaints about systemd's support for kdbus.
As a systemd committer, I certainly can. I don't have time for all of them, so I'll pull the first couple and a few other egregious ones.
> Systemd was introduced to decrease the boot up time. Now that they do not understand all implications and dependencies, let us add some artifical time we found out might work for the developers laptops. More on this small world hypothesis of the systemd devleopers below.
systemd was not introduced to decrease boot time; it was introduced to properly manage dependencies and parallelism. Such an approach happens to massively improve boot times in many cases, but that's a side-effect. The delay introduced is specifically to account for the slow/unreliable initialization of certain docking station hardware that has no other known reliable method for detection. (This is what happens in Linux with certain reverse-engineered hardware.) Importantly, this delay doesn't impact boot time, only introduces a delay before allowing the system to sleep, so even the (made up) point about systemd being about boot times isn't affected here.
> Screen brightness is something that should crash your boot up when it is not working.
The TODO item is about avoiding restoration of screen brightness at boot to such a low level that some laptops consider it to be a "backlight off" state. Someone may have shut a laptop down (even automatically) with the backlight off, but we think it should probably turn back on on the next boot. Absolutely nothing to do with "crashing" bootup.
> Systemd made kdbus non-optional in its release.
Totally made up. systemd's DBus library provides equivalent support for usermode DBus and kdbus.
> This one is a setback. Why is there no default editor in systemd in case of factory reset?
I'm not sure what this is even claiming. Is this some sort of trolling about complexity the author thinks systemd will eventually add and is some sort of advance critique?
In general, the piece shows disingenuous portrayal of actual issues to the level of clickbait and fails to understand that not everything in systemd's git repository runs as part of PID 1 (like the network management tools, for example, are a totally separate and optional daemon).
> Totally made up. systemd's DBus library provides equivalent support for usermode DBus and kdbus.
Well, it is no longer optional to compile in systemd; it can still be deactivated at run time, but the code (and checks to see which one should be used) will always be there.
No matter how many complaints you may find about SysV boot time -- or even discussion about how to improve it -- it does not make systemd's primary purpose solving that problem.
How many discussions does OpenStack's development team have about security? Is the primary purpose of OpenStack to provide security?
> Well, it is no longer optional to compile in systemd; it can still be deactivated at run time, but the code (and checks to see which one should be used) will always be there.
But that is not what the article said. It said, "Systemd made kdbus non-optional in its release." That implies that use of it is non-optional, which isn't the case at all. It is like saying Fedora has made Intel graphics non-optional because they ship support for it.
>The TODO item is about avoiding restoration of screen brightness at boot to such a low level that some laptops consider it to be a "backlight off" state. Someone may have shut a laptop down (even automatically) with the backlight off, but we think it should probably turn back on on the next boot. Absolutely nothing to do with "crashing" bootup.
Honest question: Why does an init system need to know anything about screen brightness in the first place?
Shouldn't X11 handle screen brightness?
>I'm not sure what this is even claiming. Is this some sort of trolling about complexity the author thinks systemd will eventually add and is some sort of advance critique?
This is, I think, about the fact that systemctl edit is even a thing that exists. What's the problem with ed, vim, nano, pico, emacs, etc. that necessitates some kind of built-in systemd editor?
> Honest question: Why does an init system need to know anything about screen brightness in the first place? Shouldn't X11 handle screen brightness?
I think that's a reasonable question. I am only a regular desktop user of systemd for anything with a display, so I don't have a strong opinion there. All of my advanced systemd work is on server systems; I have more opinions there.
> This is, I think, about the fact that systemctl edit is even a thing that exists. What's the problem with ed, vim, nano, pico, emacs, etc. that necessitates some kind of built-in systemd editor?
There isn't a built-in systemd editor; that's how disingenuous this piece is. Running "systemctl edit <unit-name>" invokes $EDITOR, whatever that is configured to be. Totally normal Unix behavior here.
>There isn't a built-in systemd editor; that's how disingenuous this piece is. Running "systemctl edit <unit-name>" invokes $EDITOR, whatever that is configured to be. Totally normal Unix behavior here.
Well, okay, but the suckless people are about simplicity. Adding systemctl edit seems like a completely unnecessary alias. Another feature for the sake of having another feature.
Why assume the user is too stupid or lazy to manually invoke vim and then systemctl daemon-reload?
> Why assume the user is too stupid or lazy to manually invoke vim and then systemctl daemon-reload?
This is what it does:
(1) Locates the current unit file, regardless of whether it shipped with a package or is already a custom one in /etc.
(2) If there isn't one in /etc, it copies the current one into the correct place.
(3) It opens $EDITOR on that file.
(4) It runs systemd daemon-reload.
It's really the first two steps that can be annoying because you'd otherwise have to run "systemctl status" to find where it is currently and then copy it over. I guess you could script that, but is it really so terrible to support that in systemctl -- which is just a normal user CLI utility, not anything with advanced privileges or critical impacts on system stability?
Init has no idea of the screen brightness.. that 's just another thing the author pull out from his back-passage. a separate service systemd-backlight@.service(8) saves and restores the backlight state, because firmware does not remember it or if it does, it is buggy.
Just a note as the author of the Linux Journal article, I absolutely would have mentioned Docker if I had written the article now. Unfortunately, the article only recently made it to publication (and now post-paywall) despite my having written it in January.
Our entire team will be at FOSDEM, and we'd be thrilled to meet more of the Mullvad team. Protecting systems like yours is core to us. We want to understand how we put the right roots of trust and observability into your hands.
Edit: I've reached out privately by email for next steps, as you requested.