As a *nix sysadmin since before Google was a thing, I find articles like this frustrating and arguably dangerous.
They lay out a set of "rules" the author has collated as a dogmatic doctrine that only a fool would not follow. But they provide no "why", take absolutely no context in to account, and talk only about their perceived upsides.
Anyone reading this should take each point as "here's a possible thing you should go and research yourself", because there are consequences to most of these rules.
I'm not saying anything they've written is wrong (though there is a lot of unsubstantiated opinion there), just that you should learn what any of these changes truly do before implementing them AND that your particular context is really important.
Edit:
There is a disclaimer at the top which mustn't be forgotten while reading.
> DISCLAIMER: Do not attempt to apply anything in this article if you do not know exactly what you are doing. This guide is focused purely on security and privacy, not performance, usability, or anything else.
I do appreciate people putting work into producing this sort of content, but I think the article could be improved by phrasing the various steps as suggestions and perhaps linking out to more detailed documentation on the "why" elsewhere.
Use a distribution with an init system other than systemd. systemd contains a lot of unnecessary attack surface; it attempts to do far more things than necessary and goes beyond what an init system should do. An init system should not need many lines of code to function properly.
Wow! To put such a controversial feature, and present it as "this is the truth, just how it is", without even mentioning that it's just an opinion and a lot of people have a completely opposite one, and both for good reasons. That just makes me distrust this guide because it is so opinionated and subjective.
Systemd is a security nightmare, a huge amount of code, taking over various network services ... poorly. Doesn't seem controversial at all to me. Gives me the heebie jeebies to think of systemd talking to the network before I even have the client firewall up.
Generally if security is your top priority I'd disable as much of systemd as possible. Things like find a good local resolver (like unbound) and disable systemd's implementation, etc.
I'd like to read a guide on hardening systemd. Systemd is really convenient and I hadn't realized it poses a large attack surface. A good list with possible attack vectors and how to mitigate them would be epic.
Is it really an established idea in the security community that systemd is a security nightmare?
>Is it really an established idea in the security community that systemd is a security nightmare?
It's not. The presence of CVEs in software is a sign of maturity, not insecurity. It's frustrating to see people use CVE numbers to point at some larger problem. The key element is to realize that security issues are just bugs, especially in the context of C software. What you should be cautious about is upstreams that doesn't handle security issues well. Systemd is not one of these upstreams.
Hehe... around 2002 I worked in the IT support division of a telco (South America.) At some time I suggested the use of ssh to login to our farm of Digital/Compaq/HP servers (I believe it was not included in the base OS.)
The IT Security Officer never contacted me, but convinced the management about the insecurity of SSH by showing a big list of CVEs. As a result we were instructed to continue using... telnet.
> It's not. The presence of CVEs in software is a sign of maturity, not insecurity.
It really depends, depending on both the complexity and scope of use of the project, and whether the rate of CVEs is decreasing.
Flash had CVEs regularly for years. I would say it was getting more mature, but the rate they were coming out and the fact that they just kept coming indicated to me not that the project was maturing (at least not at a rate I was comfortable with), but that either the programmers were inept, or more likely, that earlier design decision lead to very hard to reason about security and made it extremely hard to harden prior to exploits being discovered. Neither are conclusions that made me want to use that software.
OpenSSL fell into a similar situation a couple years back. Seeing all the CVEs that were coming out, once could reason "this is just a mature project", but the truth (exposed by numerous people doing audits) was that the code base and developer process was a dumpster fire.
> The key element is to realize that security issues are just bugs, especially in the context of C software.
They are, but not all people and projects produce the same quality and quantity of bugs.
All this is probably how you already see it, just not quite spelled out in your prior comment. I just thought it was important to make that distinction obvious. :)
P.S. This is somewhat divorced from the context of whether systemd is a good code base or not. I didn't intend for this to be an implicit assessment of systemd, I'm not making any judgements on it here.
Design is not a bug. Some things just aren’t designed to meet security goals. Telnet is plaintext, in most environments that’s a pretty bug security issue. That’s not a bug in the code, it’s just not designed to protect the data from tampering, evasedropping and hijacking. It just can’t operate any other way.
Configuration errors are security issues, but they are not bugs. Users can setup up things insecurely.
Human beings present their own security issues, and they are definitely not bugs you can code away.
The biggest myth about software security is that’s it’s all just bugs. This leads to after the fact thinking (well just patch it), and a huge blind spot to the fact that security isn’t something you can just build, it’s an entire process that goes way beyond just code.
I think this is mostly an issue of overloaded terms. There are security design considerations, and security issues. Telnet being plaintext is not a security issue for telnet, it's a security issue for those using telnet for something it's unsuited for. HTTP being unencrypted is not a security issue for the HTTP protocol, or an application that wants to support that (a browser), but it may be for an application that makes requests over HTTP instead of HTTPS when those requests require some level of privacy.
If an application has a design goal to be secure in some aspect, but the design they chose doesn't accomplish that, then the design itself is a bug and needs to be fixed (or they need to change their design goals). Buggy designs exist, they're the designs that don't fulfill the desired purpose.
All security issues in a the context of a project which intends to provide security in that aspect are bugs.
By that reasoning, Flash would be safe enough to stay around, seeing how many CVEs it had (1078 since the first CVE for it was published Dec 31, 2005).
It’ll be gone tomorrow though, or January 12, or later in 2021.
How would you modify your argument rather than dismiss the evidence against it?
For example, could you propose an equation using the number of CVEs and their severity along with the the rough number of known users, showing that some curve or values would indicate maturity rather than risk?
Some data for Flash[1] CVEs are available at CVE Details. Possibly you could find other products you consider safe and see how they compare.
I’d be curious if an analysis would show correlation of what you’d consider greater risk with a year that Adobe started outsourcing to a particular company or decided to EOL it; I’m not proposing that quality went down due to one of those things or that they outsourced, but analyzing the data could be useful.
You believe the security of systemd is strong, and that CVEs are a sign of use and maturity, then you’re saying ~2 CVEs per year on average is compatible with good security.
I understand you believe this, but I don’t see how it’s rational.
Note that these apply to all programs in systemd distribution (not sure how many of these are specific to systemd init program).
It seems that a lot of criticism is due systemd doing more than a classic init process does, but some of it seem unjustified since you it is not an "all or nothing" package.
For example, this specific CVE affected systemd-resolved (systemd's DNS daemon), but IIRC Debian was not affected since it did not use systemd-resolved by default: https://www.cvedetails.com/cve/CVE-2017-15908/
So it is perfectly fine for a distribution use systemd init system but use other network daemons.
>>It seems that a lot of criticism is due systemd doing more than a classic init process does, but some of it seem unjustified since you it is not an "all or nothing" package.
I hate this argument because, in practice, it is an "all-or-nothing" package from a user standpoint. Packages have a terrible habit of setting dependencies on one or two of the "modules," which wind up bringing the whole thing along with them. Sure, you can "disable" it after the fact, but then you're left with a system that will break in fun and exciting ways when something a package you requested wants isn't available.
If you don't want systemd, your only option at this point is also-ran distros.
systemd is extremely modular if you’re the one compiling it (i.e. you’re a distro-builder) but not as an end-user since it’s used as the plumbing for your distro. And since distros aren’t usually in the business of telling users not to use features they ship everything because someone might want it.
The fact that systemd bundles a lot of functionality (which is different from being a monolith) means it has a disadvantage in the terrible metric of "CVE counts".
Now if only the systemd-hating crowd would take a look at how many CVEs affected their DHCP client, nscd, nss and other resolvers, various PAM modules, syslog/klogd, ad-hoc init scripts writing insecurely to /tmp, ntpd, etc.
There it details what capabilities, and restrictions are in place for a given systemd service. From there you can edit the service to restrict even further as required if you want.
Completely agree with this. One of the biggest risks is using journald if you ask me. It makes logs fairly impenetrable if you have a forensic situation to deal with.
Now there did need to be a better solution in the init system space but systemd was possibly the worst answer that could be mustered for it.
Systemd is merely what you get when Redhat owns the development community, not the best technical solution.
Well it’s more what you do with a node that has been isolated or is suspect. Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way.
As for forwarding, there is likely a disparity between the last log written to disk and the last log received by the aggregator. What happened in that window is always interesting and always on disk on the node, unless the attacker has nuked it, which neatly leads to...
Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. Learned that from years of windows NT. you can possibly recover them from analysing disk blocks though which is even more difficult if you have journald in there again.
I prefer rsyslog, plain text and time marking. Much easier to alert on suspicious things when the mark stops arriving.
Log journaling has few practical benefits. I. Fact I’m sceptical of any from experience.
That’s fine until it doesn’t work or the journal is corrupt or you’re pulling the journal off the file system as mentioned further down the post I made. I’ve been there and the entire toolset fell to pieces in front of me.
Plain text is orders of magnitude easier and safer to deal with and easily recoverable even if only partially which is extremely difficult with journald
I mean the journald format isn’t magic or anything and 99% of what it stores is plain-text. The only issue with “corruption” is that the tooling is bad at handling it. Which is a valid complaint. But in an alternative universe people would be complaining about plain-text because less crashed on any utf decoding issue.
Basically every journald using distro immediately forwards the logs to syslog so you can just pretend it doesn’t exist and call it a day. I don’t because journald’s metadata and filtering is super powerful but nothing is stopping you from just grepping like usual.
it's a bit odd to defend the new kid on the block, journald, and then go and link to fluentbit.
I feel like we're missing the elephant in the room, which is that almost everyone is going to forward logs over the network completely bypassing journald/journalctl altogether. At which point you have to ask: who is journald for? In the server environment, you're going to send everything to Splunk or whatever.
Which just leaves desktop users to deal with the headache that is journald. Desktop/workstation users don't care about tamper detection, corruption, or any of that nonsense. If there is corruption going on, I doubt they care that their boot logs are invalid. I'm 100% sure they care more that their 3D render that took 30 hours to render is now corrupt, or some other asset or document.
And in the rare case they do need to dive into the logs, they aren't going to want to re-learn journalctl again.
journald is a solution to a 1990s problem that no longer exists.
I linked to fluentbit only as an independent journal reader, not for what it does as a service.
To be clear, I wasn't defending usefulness of sealing and signing. Just the fact that not only it's independently readable, but also you can find out about corruption of it matters to you for forensic purposes raised.
Journalctl does provide at least one significant feature though - on a properly configured system "journalctl -u foobar" gives you the logs of foobar. No more chasing which file they live in, no more stdout goes here, stderr there, and logs get split in that special way over there. This is great for desktop users.
Of the 8 voting members, at the time Steve Langasek and Colin Watson were currently employed and Ian Jackson had been previously employed by Canonical.
None of the 8 voting members were employed by Red Hat at the time.
What software actually needs to repeatedly write to these variables? Systemd itself does, but there was absolutely a world before having this option on by default. Grub and such can just unmount after they are done.
By default systemd includes journald, rather controversial (and mentioned in a comment on this thread) picked binary logging (harder for humans) and much less robust in the face of a small corruption, and ignored encryption. I believe it doesn't support encryption (by default). This replaces more robust solutions (by default) like syslog-ng and rsyslog that have long histories of battle tested real world usage.
resolved replaces the local DNS resolvers, and at least for awhile ignored DNSSEC. This is particularly bothersome since it is such a security sensitive daemon. Sure you can disable it and pick unbound, but it's not default.
timesyncd is a time daemon, replacing NTP, another security sensitive app. Not that NTP is a paragon of security, but various projects have arisen to improve things.
networkd is a replacement for DHCP and similar for IPv6. Again not nearly as nice or secure as existing solutions.
So basically decades of developer work, security audits, and competition among network services had been ditched and is now part of systemd and a typically cavalier attitude towards security.
There are good parts of systemd, it filled a need. But this swallowing of security sensitive network services really bugs me. Being able to be compromised so early in the process and the tight integration going against the idea the unix philosophy of using minimalist modular software. Normally I'd run unbound as a non-root user with access to read few directories, and write to even fewer, DNSSEC enabled, and logging via syslog (human readable). By default on an ubuntu 20.04 (and 18.04 before that) I'd have issues with dns failures... till I replaced the systemd resolver. I'd mention it to other admins and they would shrug and mention that systemd sucks.
I claim init systems shouldn't open network connections.
Sure, bind has a terrible security record, and that was part of the reason why people started writing more secure replacements like unbound.
NTP has a terrible security record, and that's part of the reason people started writing more small secure replacements like NTPsec and chrony.
Similarly sendmail's security issues resulted in improvements like postfix, which hasn't been swallowed into system yet, mostly kidding.
Linux often has multiple implementations for a given service and ease of use, performance, security and related allows them to compete. This is a sign of a healthy ecosystem and generally I think it's working well.
However systemd, by rolling this functionality into systemd subverts that system and the vast majority of systems will just accept the defaults. It also makes systemd huge and complex, last I looked there was somewhere around 5% of the lines of code in systemd than the entire linux kernel which I find scary.
systemd the init system does not open network connections. systemd PID1 does not talk to the network. `networkd` is a separate binary.
Second, systemd does not mandate using any of these components apart from `journald` and `logind`. You can pipe journald into any syslog daemon of your choice, there's a config option to do so. If you've got issues with logind I can't help you there. I don't know what it was intended to replace (consolekit, I think?) but I do remember it was badly maintained.
Third, a vast majority of deployments do _not_ accept defaults. I know that both Red Hat and Debian go for third party network managers (networkmanager and Debian's ifup/ifdown stuff) and rsyslogd was there on a default install of both Debian Buster and CentOS 8 iirc.
Finally, I repeat, systemd is a monorepo which contains many programs. Sure, you can argue about how they're tightly bound, but I can point you towards FreeBSD/OpenBSD if you'd want them to be broken up into separate repos to be more "unix-y", and if you look at systemd PID1 it's a fairly small binary which doesn't seem to offer many security holes.
Finally the biggest mistakes systemd project did is on the marketing and strategy sides.
Would have they remove the "systemd-" prefix from all side binaries and marketted them as independant projects on the website, made them usable without systemd and maybe on sub repositories, would have the systemd project just had to explain "yes we rewrite ntp, dns, etc, why not ?"
Instead they received complains about a "bloat ware" while often rewrites of industry-standard by unknown/junior people are acclaimed on HN :)
Even better, they could have integrated chronyd or others by creating "systemd integration standards" and submitting patches to these projects to gain community support and permit an easy switch to one or another implementation and let the users chose.
Though it s still already easy to use something else than timesyncd on centos at least, thanks to systemd project and distri maintainers too :)
You can't have any stable integration standard when systemd's scope is ever increasing, or paper it over with marketing. Even stuff like home directories now Must Be Improved.
It isn't comparable with typical rewriting of industry-standard by newbies.
thank you for the follow up. You have made some good points but I couldn't find any explanation to what you said : systemd talking to the network before I even have the client firewall up.
Worth noting that these are optional. You can forward logs to your preferred syslog daemon. You can turn off resolved, timesyncd, networkd and use something else instead of them.
It's not ideal when looking at the whole environment, but specifically for our services - we've got the choice.
Systemd's scope is increasing over time, most recently with handling system /home directories. Seems very possible that the ability to disable features in the future versions of systemd will be removed because they do not have the required integration into systemd's view of the world.
In any case it's the default behavior of systems that should be secure and systemd is the exact opposite of that.
I see, I've heard about it but I couldn't find any reference that systemd starts listening on sockets before the firewall service is started. Even if it did, with systemd you can setup your firewall service to start before the eth link goes up.
You can't typically start the firewall before the eth link goes up, since rules that contain network interfaces might not work before those are created (e.g. in case of bridge or bonding interfaces). Systemd's "start everything at once, let the dependencies sort out automatically" is a major regression for server systems. With sysv-init you could just pick the right ordering and be done, everything was stable for every boot after that.
You can, using the directives "Wants=network-pre.target" & "Before=network-pre.target". This will make sure the service is up before any network interface is configured.
Not quite. First, this doesn't apply to any interfaces that are created by system services like VPN tunnels. Second, you would usually want and need network-online.target, because firewall config often is applied after network-pre only. Third, dynamic stuff like VLAN interfaces also only appear after network-pre and often only after network-online. So no cookies for you. This kind of non-attention to details is actually what makes systemd bad: Everything looks like the authors laptop, if it doesn't, tough luck.
Edit: Sorry, I just realised what you mean. I don't know about iptables, but I've got nftables set to come up before the network is set up and it works perfectly fine without enp3s0 or wg0 having come up. The rules are set and then the interfaces come up.
Original comment follows:
I'm... not sure I understand. VPNs would only work after your physical interfaces are up, right? So if you want your firewall rules to be applied before a VPN interface comes up then you'd be perfectly fine with network-pre. From the page I linked in the comment you replied to:
"network-pre.target is a target that may be used to order services before any network interface is configured. Its primary purpose is for usage with firewall services that want to establish a firewall before any network interface is up. It's a passive unit: you cannot start it directly and it is not pulled in by the the network management service, but by the service that wants to run before it. Network management services hence should set After=network-pre.target, but avoid any Wants=network-pre.target or even Requires=network-pre.target. Services that want to be run before the network is configured should place Before=network-pre.target and also set Wants=network-pre.target to pull it in. This way, unless there's actually a service that needs to be ordered before the network is up the target is not pulled in, hence avoiding any unnecessary synchronization point."
So ideally you'd run your VPN service after network-online.target has been reached, which would certainly be after the firewall rules have been applied.
The great thing about persistent interface names is that if you know the names of the interfaces which are going to be coming up, then you can set up a firewall before any of them come up. Which takes care of dynamic VLAN interfaces. You can use IP addresses or blocks to refer to them in your firewall config, am I right?
If you think I'm unable to grasp your situation, can you give me a more detailed example?
> Systemd's "start everything at once, let the dependencies sort out automatically" is a major regression for server systems. With sysv-init you could just pick the right ordering and be done, everything was stable for every boot after that.
This is a big point. All of the systemd systems I've been exposed to in the wild have non-deterministic inits, i.e. if it takes longer than average to boot up once (A problem I had a few times), starting it again doesn't replicate said problem. If the network craps itself when booting sometimes, that problem is variable so it's impossible to tell if you've fixed the problem, and it was what you suspected, or if it's something else and you only think that you've fixed it.
On the other hand, my alpine system consistently boots up. If it errors once, it will error consistently enough to allow tracing of the cause, but regardless, it has never had a problem on boot.
iptables and nftables both have ways to declare rules for interfaces that come and go. Identifying dynamic interfaces is done by string comparison instead of by index (for static interfaces).
Which doesn't always help you if you are actually using one of the make-firewalling-easier-daemons (which I personally would advise against). Those often only support very basic features, but are pushed heavily by the commercial distros.
I have yet to see solid evidence it is any bigger of a nightmare than a mess of bash scripts that was what came before, but ok.
> taking over various network services ... poorly.
The networking features of systemd are entirely optional. If you don't know how to disable them if you don't want them, the right course of action is not to blame systemd.
People who grew up on SysV init like to shit on systemd as it erases their existing knowledge, but to pretend that SysV init wasn't a total, inconsistent mess or that anyone was maintaining consolekit before logind came around is distorting the truth, to put in mildly.
“systemd contains a lot of unnecessary attack surface; it attempts to do far more things than necessary and goes beyond what an init system should do.”
I’d want the author to explain this further. Perhaps they could tell the admin of the system how to reduce the attack surface, detail what services it should not be providing (and why), what services it provides that are worse than the alternative, and how to turn them off or harden them.
“An init system should not need many lines of code to function properly.”
That’s a ridiculous statement. Lines of code is an absurd measurement to determine the security of anything.
Systemd provides a lot of network functionality in systemd-networkd, journald, timesyncd, etc. that is remote attack surface. All the systemd "cloud of daemons" is tightly coupled by dbus interfaces that enable an attacker to move from one exploited system service to the next. Even if the attacker doesn't manage to find an exploit in another system service, DoS is easily possible because the DBUS interfaces are quite fragile. Even as a benevolent admin it is easily possible to get the system into a state where e.g. clean shutdown is no longer possible because systemctl doesn't want to talk to systemd any longer and you cannot fix that. systemd-udevd also has raceconditions galore, so sending any message to it in the wrong order relative to another one will kill the system, maybe even open exploit vectors. At the very least I would, for hardening, recommend not using any network-facing systemd functionality.
And lines of code are not ridiculous, they are the best first-order estimate available. Of course an actual inspection of the code is better for a comparison, but that is a huge task. sloccount is quick and easy.
> Systemd provides a lot of network functionality in systemd-networkd, journald, timesyncd, etc. that is remote attack surface.
Everything systemd does is something the OS was already doing before, just using a bunch of independent-upstream-project daemons glued together with RC scripts. That attack-surface existed before. That LOC-count existed before. It just wasn't monolithic under one project/org, and so wasn't being perceived as a problem.
The various interactions between daemons, also weren't getting audited during upstream-project security audits, because all the interaction potential only existed in the downstream distribution, rather than being part of systemd's monolith + integration test-suite.
> DBUS interfaces are quite fragile
What, and what it replaced (arbitrary ug+w sockets in /var/run, speaking to custom one-off RPC-listener ABIs for each daemon) wasn't fragile?
IMHO, all that's changed is that there's more uniformity, meaning a single set of tools can be used to test+harden all such RPC interfaces, and security flaws can be fixed in one place (DBUS itself) rather than requiring theory-level descriptions of vulnerabilities to be communicated to each project, whereupon they're hopefully translated to separate one-off patches the affected daemon.
> systemd-udevd also has raceconditions galore, so sending any message to it in the wrong order relative to another one will kill the system, maybe even open exploit vectors
100% true of what came before it. Hell, some of my Linux systems from the 90s and early 2000s would lock up hard just because I `modprobe`d a webcam driver when the webcam was already USB-connected.
> What, and what it replaced (arbitrary ug+w sockets in /var/run, speaking to custom one-off RPC-listener ABIs for each daemon) wasn't fragile?
Sockets like initctl are far less fragile. You don't need a dbus-daemon that you can't restart if you cant talk to systemd. Your parser needs to recognise single-character commands instead of a complex rpc syntax plus an xml parser or javascript vm for filtering. You can't honestly believe DBUS isn't miles from the robustness of initctl.
> IMHO, all that's changed is that there's more uniformity, meaning a single set of tools can be used to test+harden all such RPC interfaces, and security flaws can be fixed in one place (DBUS itself) rather than requiring theory-level descriptions of vulnerabilities to be communicated to each project, whereupon they're hopefully translated to separate one-off patches the affected daemon.
Instead of easy-to-audit socket permissions you get a complex mess of xml and javascript filtering rules. This isn't progress in terms of security, its quite the opposite. Pounds of additional attack surface and additional obfuscation hindering defense.
What systemd gives is structure + consistency; service definitions are declarative and knowledge distro portable.
To me that that is the big benefit of systemd over sysV init.
> Pounds of additional attack surface
Again, show me a major in the wild exploit that abuses these supposed 'pounds of attack surface'. As was said before, systemd doesn't actually have that much new surface that wasn't there before. It's just before it was not maintained in any sort of organized manner but rather was scattered across tens of different projects in various state of no longer being actively maintained, as I said consolekit is a perfect example.
systemd is actually pretty modular. You don't have to run the components that are not useful to you thus limiting the attack surface greatly, if you're actually concerned about it.
I know some people have an allergic reaction to Poettering, but the guy's willing to touch some of the rotten parts of the ecosystems few others dear.
> People who grew up on SysV init like to shit on systemd as it erases their existing knowledge...
Painting people who doesn't like systemd like old, beardy creatures afraid of learning new stuff is extremely simplifying the situation if not being misleading.
My personal opposition about systemd is about binary logs and going against "doing one thing, well" principle. I may rant it all day long about the things I don't like about systemd and praise its useful features for a similar amount of time but, I'd never bring up "I learn to everything from ground up again!" as an issue.
Software evolves, everything evolves. We need to adapt regardless of we are system administrators of a big farm or a user of a measly Linux system.
> My personal opposition about systemd is about binary logs
syslog-ng supports the systemd journal natively so you'd never see a difference from before if you don't want to.
> doing against "doing one thing, well" principle
The principle is dogma that has held back Unix imo. It's useful for simple CLIs but that is about it. By your definition the kernel itself goes against the principle.
systemd is a systems manager, it takes care of it during its entire lifecycle, keeping home directories portable is one thing it can do for you that is entirely optional but very useful, managing containers is another. These are however separate components - homed and systemd-nspawn respectively. It is not all rolled into a single 'systemd', this is a misrepresentation.
systemd is very modular. People like to paint it as a monolith because it lives in a monorepo for the most part, but it is not actually built as one and individual components can be disabled or swapped out.
> syslog-ng supports the systemd journal natively so you'd never see a difference from before if you don't want to.
We use rsyslog with systemd and it works too, however it's just another level of abstraction so, I'm not fond of piling stuff over and over. Binary logs' and journals' usefulness can be also debated. It of course brings some advantages to the table but, it doesn't enable anything groundbreaking for me.
> The principle is dogma that has held back Unix imo. It's useful for simple CLIs but that is about it. By your definition the kernel itself goes against the principle.
Actually I don't consider it dogma, because it has some very important results and corollaries. This principle reduces secondary complexity (glue logic, complexity required to do many things with one code base, etc.) greatly hence the software can be relentlessly optimized. grep, find, sed, awk, cut, head, tail, less all are extremely performant tools. I develop high performance scientific applications and making a tool accomplish more than a couple different functions either reduce performance (generalized algorithms) or increase size a lot (specialized functions for everything). Its refactoring, verification, maintenance gets complicated and expensive fast.
UNIX pipes and small optimized tools are much more useful beyond simple CLI commands and small scripts. Since the tools are fast, low memory footprint performance behemoths, you can build very fast and reliable machinery with them. You can even do data mining with them [0]. We have some impressive stuff running under the hood in our cluster which only use standard GNU utilities.
Also, tools like rsync and rdiff can lift much more than their size. They're proverbial ants of data transfer and they also do one thing well. Same thing can be said for vi and nano too for text editing. Similarly, modern tools like Atom still shudder and die with a single big file while using extreme amounts of memory. OTOH, vi doesn't even sneeze with files thrice as large.
Kernel doesn't breach this principle either. It provides an interface between the hardware and software & lives in its own space; that's it.
> systemd is a systems manager, it takes care of it during its entire lifecycle, keeping home directories portable is one thing it can do for you that is entirely optional but very useful, managing containers is another. These are however separate components - homed and systemd-nspawn respectively. It is not all rolled into a single 'systemd', this is a misrepresentation.
The problem with systemd is not purely technical. Also, I want to be clear that I'm not a die-hard systemd critic. The technical side can be summarized as "Hey! This thing is complicated, developed very fast and can bring some (stability and security) problems back during its teething. Please be careful". Other side is social and can be summarized as "Hey! You're developing this, but you're not listening to us and pushing your opinions down to us using your power. This dangerous in every aspect. We can develop this together to something better but, you don't listen to us.".
Also I want to remind that, parallel sysV-init was pretty fast machinery and was very manageable.
I'm aware that systemd is modular. I'm not using more than half of it right now and I'm happy this way. One of the problems with systemd is opaqueness. When systemd overtakes a part of the system with its module, it's very hard to discover it. Disabling the module and using an older or alternative approach is also not straightforward sometimes. As a result, systemd feels like an overzealous octopus with no indication of its intention. Holding everything it can reach with a death-grip and requiring some serious strength to pry it off.
With a better communication and feedback loop, we'd be in a very different place. Maybe systemd would be the same systemd but, without the unproductive rock-throwing, flaming & shouting.
Linux is evolving like every other software or anything in life, but some ground principles are much more valuable than they seem. Throwing them away and labeling them as dogma just because they're old is considered harmful and dangerous.
> I want to remind that, parallel sysV-init was pretty fast machinery and was very manageable.
That does nothing for the fact that SysV init was a pile of shell scripts inconsistent across distros and even individual packages. I'd take systemd's declarative, portable service definitions any day over that mess.
SysV init did not take care of consolekit being an unmaintained mess either. systemd did.
Besides, systemd can recognize that I had plugged in a specific kind of device that needs this sort of setup and do that for me automatically. Hence the 'manager' part. None of its supposed 'competitors' can do that themselves.
> Kernel doesn't breach this principle either. It provides an interface between the hardware and software & lives in its own space; that's it.
I don't know, eBPF for example is an entire complex beast of its own that could be argued to violate that, well beyond what's 'required'.
If you take the view that as long as the system has a single defined job, (interfaces to hardware and exposes APIs to userspace for the kernel), systemd does have a definition like that too. It manages the system's dynamic resources during its lifetime. That's a 'single' job. It's not if you decompose it, but the kernel is in a similar situation at that point.
> When systemd overtakes a part of the system with its module, it's very hard to discover it.
systemd does not do that, sounds like you're blaming bad distro defaults (optional components being enabled by default) on systemd.
> That does nothing for the fact that SysV init was a pile of shell scripts inconsistent across distros and even individual packages.
I've never seen the inconsistencies, sorry. All the service files I've written were greatly portable around what I've used so far.
> SysV init did not take care of consolekit being an unmaintained mess either. systemd did.
No other service should hide the shortcomings of other services. This sounds like WoW's extreme tricks or Kubernetes' dockershim layer to keep compatibility. Now, doing this is wrong.
> Besides, systemd can recognize that I had plugged in a specific kind of device that needs this sort of setup and do that for me automatically. Hence the 'manager' part. None of its supposed 'competitors' can do that themselves.
That's the point. If that task can be handed over to another daemon (like udev) for setting it up, why systemd assumes that setting it up is its job? I'm not trying to say that doing this is flat out wrong but, when asked about why, getting "we're doing it , because why not?" as an answer leaves a bad taste in the mouth.
> systemd does not do that, sounds like you're blaming bad distro defaults (optional components being enabled by default) on systemd.
I think I failed to convey what I tried to say there. I wanted to say that there's no easy way to see whether a feature is managed by systemd or not and hence systemd tries to "self-heal" things sometimes, system management becomes a tug-of-war until one understands that systemd is managing that. If there was an easy way to understand that, it'd be an easier path.
All in all, I personally don't oppose what systemd brings to a table and see the inspiration from macOS' launchd. However, the main thing I strongly oppose and criticize is the blind egg-throwing to other init systems, the overzealous
protection of systemd and avoiding discussing its potential shortcomings and problems altogether.
I want to reiterate that I'm not against change and evolution, I'm against forcing it with a stance of "we, only we know the best. now shut-up and use it!".
I've seen plenty to convince me that having a full shell scripting language and not a very good one at that as your service definition language is a terrible idea.
> No other service should hide the shortcomings of other services.
I don't think you get it. It is not hiding the shortcomings of other services. It is implementing its own service because the other one was a pile of unmaintained crap. But I guess that kind of security risk is OK as long as it's a separate repo? What a weird dogma.
> That's the point. If that task can be handed over to another daemon (like udev) for setting it up, why systemd assumes that setting it up is its job?
It does hand it over to udev, but someone needs to notify udev stuff is happening. systemd has easy access to that information so it does the job. Also, you do realize that the people who maintain udev are the same who maintain systemd right? Again, because it was a hard job and nobody else, including other init systems, were willing to pick up the slack.
> what I tried to say there. I wanted to say that there's no easy way to see whether a feature is managed by systemd or not
I understand that but imo that should be on the distro to communicate. On Arch for example, the wiki always gives a clear indication of whether a thing like DNS can and is managed by a systemd component or not.
A `systemctl component list --status=enabled` would perhaps be a nice addition, agreed.
> I strongly oppose and criticize is the blind egg-throwing to other init systems, the overzealous protection of systemd and avoiding discussing its potential shortcomings and problems altogether.
It's funny you say that because as a very early adopter of systemd I saw things up close and it was the exact opposite. People were literally threatening Lennart Poettering with violence for writing a (good imo) piece of free software. That he decided to stay on the scene despite that is honestly remarkable.
But as for other init systems, I don't doubt there's some good ones, sysV init wasn't it, but the main problem is that all the others seem only interested in the init part and are happy to depend on rotten parts of the ecosystem. Unlike systemd, none of them were saying; 'ok, consolekit is unmaiintained, let's pick up the slack on session management' (maybe in a separate repo if you insist) or 'oh, udev is struggling, let's pick up the slack' - no they were like am going to do init and hope these other pieces don't come crashing down like a pile of bricks.
Also, I've started using systemd-homed to make my setup portable across my machines. None of the supposed competitors of systemd offer that either.
I like to compare it with PulseAudio; people suggest to me all the time how OSS, ALSA etc. are 'good enough' etc. but I have a fully FLOSS wireless audio house setup right now thanks to PulseAudio, none of the supposed 'competitors' offer that, so they're not competitors for me, (PipeWire eventually might by literally re implementing all of Pulse).
bayindirh is raising valid points of criticism, which I think you can tackle without trying to raise a stink.
The point about scope e.g. isn't "weird dogma" imo, but very valid, because it has security implications for an init system.
The points you raise about unmaintained alternatives is also valid, but doesn't invalidate his criticism.
I understand you feel that systemd has been treated unfairly before, but I suggest you shouldn't try to match the level of spite and throw it the other way, especially when someone is very nicely bringing up what they are worried about.
> I've seen plenty to convince me that having a full shell scripting language and not a very good one at that as your service definition language is a terrible idea.
If we look enough, we can find convincing examples about Python being very backwards and brainfuck being the best language in the world. I had a friend who tried to solve every problem in OCaml. Another one thought that Lisp is the one and only way to do anything with computers. If there's one important thing I've learned over the years, it's the harmful nature of being fixated to ideas. Again, I want to re-iterate that I'm no enemy of systemd. I use it everyday on a 1000+ server fleet. I also used sysV-init on the same fleet. Both got the job done, albeit differently.
> I don't think you get it. It is not hiding the shortcomings of other services. It is implementing its own service because the other one was a pile of unmaintained crap.
Yep, I didn't get it. I missed when systemd reimplemented/replaced consolekit. I thought it shadowed it and hid its shortcomings via intervention, my bad, honestly. When systemd started to get adopted fast, I was working literally underground in a time sensitive project, isolated from outside world. I spend ~2 years without being able to track this stuff closely.
> It does hand it over to udev, but someone needs to notify udev stuff is happening.
I thought there were external mechanisms which notified udev reliably. I remember triggering it for a USB flash drive related event, intertwined with PAM, some network services and other inputs.
> Also, you do realize that the people who maintain udev are the same who maintain systemd right?
Nope, I was buried underground, cut-off from outside world. Again, it's one of the parts I've probably missed.
> Again, because it was a hard job and nobody else, including other init systems, were willing to pick up the slack.
Other init systems didn't notify udev because they thought that it's not their job but another deamon had to do it, because other so-called competitors only care about services. Hardware, hostname, time, resolver, containers, home folders, and the kitchen sink shall have their own daemons they've thought probably.
> It's funny you say that because as a very early adopter of systemd I saw things up close and it was the exact opposite. People were literally threatening Lennart Poettering with violence for writing a (good imo) piece of free software. That he decided to stay on the scene despite that is honestly remarkable.
I'd never threaten anyone about the software they write or the way they behave. Again, at that time I was underground so I had no chance to see it. I also don't support anyone doing this to anyone, for any reason. However, if one of my fellow developers got banned from submitting patches to kernel because they're not fixing their own mistakes and kicking everyone like a spoiled goat, I'll definitely have a good talk with him about it. Using fame to blame others is not the good way to do it.
> But as for other init systems, I don't doubt there's some good ones, sysV init wasn't it, but the main problem is that all the others seem only interested in the init part and...
Pardus' "Mudur" was similar to systemd in many aspects however, it was so integrated that it was not possible to install it part by part. It was faster than both upstart and parallel sysV-init ever be. IIRC Ubuntu wanted to use it, but adopting it was too disruptive and they gave up the effort.
> Also, I've started using systemd-homed to make my setup portable across my machines.
If that fits the bill, that's nice. I don't need portable homes across my systems (all of them are intentionally setup differently due to reasons and preferences) and wouldn't use it for now. If I decide to use it, I'll enable it but, I'm fine with the old method now.
> I like to compare it with PulseAudio; people suggest to me all the time how OSS, ALSA etc. are 'good enough' etc. but I have a fully FLOSS wireless audio house setup right now
PulseAudio brought some needed things to Linux audio, but it needed a lot of hammering in the head to bring it to this level. Initial implementation was a bit boneheaded in behavior. Its automatic umpixing of stereo streams to multichannel still doesn't make sense in most cases. So it needs more sensible defaults from the source tree.
Also, you use the word competitor in a rather aggressive and provocative way. I don't know whether that's your intention but, we're just conversing here. I'm not angry or something or I'm not trying to sabotage the projects you like. I'm just telling the things I see as wrong and try to make some constructive criticism. I may be failing at doing it in an excellent way, but I'm no way hostile to you, or the projects in general.
You know? There were init systems before systemd, working differently than 'with bash scripts', and working very well. They still are available, still working very well, still being developed, still adapted, still lean and mean.
This dichotomy systemd vs. SysV init is simply false, and only applies to the fortgeschrittene Fertigfutterfresser who are slaves of shrink wrapped anything ready to run.
Like obese McDonaldians, trying to discuss gourmet-food :-)
Were these other systems willing to pick up the slack in maintaining consolekit or coming up with their own solution? Didn't look like it. What about udev? Why did systemd devs have to pick the slack on that too?
Is is perhaps because the other init systems are only willing to touch a very narrow definition of init while depending on rotten, unmaintained pieces of the ecosystem like consolekit without being able/willing to maintain it themselves?
Or perhaps the plan is to let systemd folks do that while shitting on it so we can cook up a 600 LoC 'minimal init' while outsourcing a whole lot to systemd maintainers while taunting our minimalism?
You mean to say me that there’s no such thing as “superior programming practices”./s
I’d take a heavily used and tested system over one programmed “with superior practices”, like obsoleting rarely used code, any day of the week.
Yes, heartbleed was bad, but those type of bugs are rare, and once made public patches come fast. You don’t make your system more secure by exchanging the rare occasional general bug with obscure and unknown bugs (with unknown reach).
A lot of people thinking in security tend to fall for the availability bias, they want to protect themselves from the bugs that make news, while ignoring the ones that are removed day by day in a changelog of some minor utility they are using.
> heartbleed was bad, but those type of bugs are rare
Not really. There's a constant stream of security vulnerabilities arising from buffer-overflows and memory-management bugs, even in the most high-profile C/C++ codebases like the Linux kernel and Chromium. C and C++ are minefields for undefined behaviour, and many security vulnerabilities can be tracked back to instances of undefined behaviour. Rewriting in a safe language, like SPARK or the safe subset of Rust, would close the door on these vulnerabilities.
That's not to say I'd uncritically jump aboard a safe-Rust replacement for OpenSSL. As you say, there's much value in mature and battle-tested code, and Rust's safety guarantees wouldn't guarantee you a bug-free SSL library, they only guarantee the absence of certain kinds of errors. Even a fully formally verified implementation in SPARK could still have side-channel vulnerabilities (e.g. timing issues).
Bugs are frequent, but most of them have limited scope, bugs that break everything from computers to smart toasters are rare.
As a C programmer myself I’m well aware of how big the mine field is, and I’m a big proponent for validation in the Rust style, but that was not the point.
This guide advocates for replacements for programs and libraries written in C++, with most of those replacements being written also in C++, but with “superior programming practices”, even when those replacements are very rarely used in most environments.
Even there it depends. 'Programming practices' is vague. Even C can be tamed, at great expense, using formal methods techniques. [0][1][2][3] Adoption of such methods can give a solid assurance of the lack of UB, like use of a safe language. Weaker measures, like adopting MISRA C, don't provide such strong assurances (although they can eliminate certain categories of errors), and as you indicate, their real value is a bit more subjective. Mandating a bad programming style could actively make things worse.
Agree. Not something I know a lot about but it seems to be a significant undertaking. I figure a production-grade implementation in safe Rust is more likely than a verified implementation in SPARK.
I don't know how serious the rustls implementation is. Nice to see it makes no use of Rust's unsafe features.
I stopped reading after that. I understand the author has good reasons from his perspective. I m also realizng the need for opinionated tech solutions in order to create a cohesive system. However the authors opinion is not moving towards a direction I want to follow.
Luckily (or perhaps unluckily for you because you stopped reading) there are plenty of systemd hardening tips in the article. Its quite strange to stop reading after an init system opinion but these init system religions are nothing if not strange.
One of the things I try to do with my tech choices is watch my delta from mainstream options, where I expect most development and bug fixes happen. Given that systemd has been adopted by the bigger linux distributions I see more value in using it and have the big vendors fix any problems than trying to use niche solutions.
I m surprised that the author did have systemd hardening suggestions, didn't expect that. I 'm tempted to go back and check them.
When I see stuff like that I usually just stop reading and move on knowning it will eventually get deleted as I delete history over 6 months old in my browser
My first mistake as a green "jack of all trades" admin was getting scared and feeling like I had to apply a "Hardening guide". Wasted so much time and energy because I didn't know how to determine what was a priority, both when it came to business needs and security. Guides like this presume you have generous amounts of time to put into a
Linux hardening project with a questionable ROI when the person looking at it might be a solo overwhelmed admin in a security insensitive line of work.
Fact of the matter is that you can't take a broad checklist approach to security because you can't secure everything, you have to have an idea of where your biggest risks are and focus on those. If the computers in question aren't dealing with PII which one has some moral duty to protect, security risks can be outweighed by other business risks, and going on a security crusade can INCREASE business risk. Sometimes things that might be security risks are mitigated by other aspects of IT security or non-IT controls making them very low priority. Sometimes hardening certain things will have an especially adverse not worth it effect on certain users/applications (which is why these hardening settings aren't defaults).
Maybe worry about phishing attacks, that your authentication strategy makes sense, that your firewall is intact before worrying about arcane aspects of Linux Hardening. A guide like this is fairly useless in the real world without some discussion of the risk/benefits of hardening something and some mechanism to prioritize. To me this guide serves as an interesting starting point to constructing a practical hardening strategy, but I would advise against following it blindly. If you don't know what you're doing I would trust that Linux has fairly sane defaults and focus elsewhere to start with. Configuring your systems in non-standard ways is a Pandora's box of headaches.
A lot of this hardening is based on a fairly obsolete Unix view of the world. E.g. preventing users from seeing other users' PIDs, changing /etc/securetty, non-root X server... In real life, nowadays, a single system is likely to be running a single application, and once that's owned there's very little of the rest of the system to take over.
Owned a Web app with access to user accounts? That's enough to steal all the credit cards flowing through it, and spy/scam all the users in it. Becoming root, or peeking into other users, is usually unimportant because there's nothing special happening there.
I guess Kubernetes is the one thing that sort-of brings back these concerns, but even that has its own more modern particularities (e.g. the guide doesn't even mention cgroups).
I mean hell, in a corporate network where you might want to look at hardening the configuration you roll out to your machines, you're more likely to have a hardware firewall and a VPN requirement before you even get to your "squishy" Linux box.
Yes, and there's a clear mix of threat models (for instance, sometimes the guide seems to be emphasizing Whonix-style user anonymity, but other times something else entirely).
I see some very interesting things to go learn about in this guide, but as you note, it's not really presented that way. I would find it more useful if it either picked a more specific threat model and stuck to it, or suggested a particular system that has already made opinionated choices on some of these protection mechanisms (maybe like Whonix, Tails, or Qubes), or invited a discussion of "for each of these options, why isn't it already the default everywhere? did some people find the associated threats too exotic, unlikely, or irrelevant? are there harsh tradeoffs here?".
One example for me is the ptrace restrictions. It's clearly true that attackers can use ptrace to attach to a running process to steal secrets (if you are relying on processes as a security boundary, this feature will tend to pierce it), which was the main motivation for the kernel having an option to restrict it. On the other hand, if you're a software developer or security researcher dealing with binary executables, ptrace is very helpful to you and you may be irritated if you turn it off. That might be a sign that there should be different OS default profiles for "R&D" and for "production server" or "dedicated administrative workstation" (or maybe not), and perhaps if you don't know what ptrace is you should default to disabling it, but it still strikes me the wrong way not to see any context about this.
I agree with this and would add that some of the hardening they are suggesting will make debugging problems much harder. This article assumes that if you are doing this in a corporate or datacenter environment, that all of your engineers have vast experience. Even then, some of the tweaks mentioned will require making changes and rebooting to debug some problems.
Not only should people read up on and understand the implications of each change, but also do "battle testing" before doing this in production. That means going beyond functional testing you might get from a CI/CD environment and instead letting this bake in a staging environment for a few months first. It needs to be an environment with real traffic, real workloads, but low service level agreement and low revenue impact for an extended period of time.
As an aside, I can tell they have not tested some of the things they are suggesting. The first obvious one that pops out is the changes to /proc in fstab. That won't do anything on 100% of linux distros, as /proc gets mounted by the init process. That has to be done in a init script as a "remount". Also, the "ipv6.disable=1" in the kernel options is not advised. Do this in sysctl and in the network scripts, or some things may break depending on how your system is being used. The ipv6 module is used for more than ipv6. There are also some kernel options that change in kernel 5.x that this article does not mention. The ulimit settings will not apply to systemd processes and will not apply to root. They mention disabling debugfs, but not binfmt. If I had to pick a more dangerous thing to disable it would be binfmt, which brings in dangerous concepts from Windows 95/98. I need to stop reading this or I will spend all day writing notes on the gotchas.
Would you tell me why the ipv6 module used for more than ipv6?
If you don't want something running, disabling it at the kernel level makes sense to me, so I am curious as to why that is not the case. If things don't work with it disabled in the kernel, but DO work with it disabled via sysctl, what is actually being disabled?
One example would be other kernel modules that require it as a dependency. So if you disable loading the ipv6 module, you can no longer load any module that references it, even if you would not have been using ipv6 in that module. The first one that pops into mind is the sctp kernel module. This also used to prevent loading the bonding module but I am not sure if that is still the case. You might try testing that.
Newer kernels have the ipv6 code built into the kernel, so the kernel boot option to disable ipv6 won't do anything regardless. The supported way to disable it is sysctl and the networking scripts.
To be fair I can't blame the author. The why part probably doesn't exist because he wrote it for himself, or, he wrote it for people who he knows and would be able to seperate what advice to follow for their use-case.
This is just an assumption on my part, but I feel like he just wrote down a list of what he has followed down in the past. He didn't write down the "why" because he wrote it in context of his own experiences.
Like that systemd portion looks very icky, but when you have the disclaimer in context. You'd want to go to something with the least amount of attack surface. Systemd is a giant project, and a very useful one at that, and for context I am firm supporter of systemd, but I can see why he'd put that in the guide, since his disclaimer says the guide isn't really about usability but minimizing the attack surface.
In turn it got submitted to hackernews by someone who read their article, and unfortunately some people would treat it as gospel and turn off their brains, since as you mentioned there's no "Why?", but it makes sense to not include it if he didn't really write it for the masses but just to document something for himself and his friends/colleagues.
We need to be better at being suspicious about any information. And do our own research and experimentation. But who has the time for that in this day and era when information is expected to be received by push notification and most people can't comprehend messages longer then an SMS.
A lot of what they talk about also conflates security with anonymity. While it may be good advice for connecting to coffee shop WiFi on your laptop, for a secure network, being able to tell who performed an action is less of a security hazard and more like a security requirement.
Your disclaimer applies to every article ever. I agree 100% but there’s nothing really specific to this article that makes me worry more. Its just common sense.
They lay out a set of "rules" the author has collated as a dogmatic doctrine that only a fool would not follow. But they provide no "why", take absolutely no context in to account, and talk only about their perceived upsides.
Anyone reading this should take each point as "here's a possible thing you should go and research yourself", because there are consequences to most of these rules.
I'm not saying anything they've written is wrong (though there is a lot of unsubstantiated opinion there), just that you should learn what any of these changes truly do before implementing them AND that your particular context is really important.
Edit:
There is a disclaimer at the top which mustn't be forgotten while reading.
> DISCLAIMER: Do not attempt to apply anything in this article if you do not know exactly what you are doing. This guide is focused purely on security and privacy, not performance, usability, or anything else.
I do appreciate people putting work into producing this sort of content, but I think the article could be improved by phrasing the various steps as suggestions and perhaps linking out to more detailed documentation on the "why" elsewhere.