It took me a lot of reading before understanding that this is an OS on which you run a Kubernetes node//master and not a so-called OS on top of Kubernetes.
Somehow When I read "OS for Kubernetes" I thought it was yet one more layer of abstraction on top of Kubernetes.
I wish these sales-type pages had a "click here if you're an engineer." The page would contain a buzzword-free, straightforward explanation of what the product's purpose is, what its major features are, and what interfaces are exposed for control and flow.
I have passed on countless products when evaluating solutions for having very flashy sales pages with no actual description of the product itself. "Improve your ROI! Decrease Downtime!" aren't features, per se, but potential benefits that can only be determined if the product is a fit, and I can only determine fit if I know that the hell it does.
As an engineer I understand this completely. We are already discussing internally how we can improve our site and documentation. Our goal is to create a vibrant community and getting all the sales fluff out of the way is something I support!
Hey folks, Talos creator here. Happy to answer any questions you guys may have. Sounds like some confusion about exactly what Talos is. A lot of good feedback here that we will take and improve our documentation.
Talos is a Linux distribution built specifically for Kubernetes. The short version is that we have stripped out absolutely everything that is not required to make a machine a Kubernetes node, including SSH and console access (I will explain why).
Here goes the long version. We have done a number of things to improve security, including a read-only filesystem, except for what the Kubelet needs (/var/lib/kubelet, /etc/cni, etc.). It runs entirely in RAM from a Squashfs, and only Kubernetes makes use of a disk. We have stripped SSH/Console access and added a gRPC API that gives engineers the ability to debug and remediate issues.
We didn't just stop at this. We are writing everything, including the init system, in Golang, which allows us to integrate deeply with Kubernetes. Everything about Talos is API driven.
Some of the highlights include:
- SSH/console access replaced with gRPC API that is secured via mutual TLS.
- Immutable. Immutability prevents drift, making the cluster consistent across the board.
- Automated upgrades that can be orchestrated in an intelligent way. By using Kubernetes events, and our API, we can roll out upgrades from an operator (currently a WIP and planned for release in 0.3 Talos) and do safe in a safe manner.
- Cluster API (CAPI) integration that allows rapid creation of Kubernetes clusters using Kubernetes style declarative YAML.
- Support for AWS, GCP, Azure, Packet, vSphere, Bare Metal, and Docker. The experience for each is consistent, making it easy to reason about Talos regardless of where you run it.
- CIS and KSPP security configuration enforcements.
- Keeping current by supporting the latest and greatest version of Kubernetes, while writing upgrade paths into the system.
- Support for local Docker based clusters, easily created using our CLI. This is super useful for creating CI pipelines where you might want to run integration tests against the same Talos/Kubernetes versions running in production.
- Installs and upgrades are performed via containers.
We feel that by removing SSH/console, making the core of Talos read-only, and treating the nodes as ephemeral machines, we are creating a much more secure way to run Kubernetes. A really good talk was given on these ideas at Blackhat this year: https://swagitda.com/speaking/us-19-Shortridge-Forsgren-Cont.... We feel we align with the recommendations made there.
In addition to security, we envision a system that will be self-healing and intelligent. By having an API and integrating with Kubernetes, the sky is really the limit on the tooling we can build to create this self-healing system.
Our goal with Talos is to allow engineers to more or less forget about each individual node. Managing the OS alongside Kubernetes is a lot of work.
I will address the questions and comments as replies. Feel free to ask more as a reply to this comment.
The automation here is predicated on CoreOS distributing matching { kernel, headers, toolchain } artifacts for each release, and in particular how specific OS releases get promoted from the alpha -> beta -> stable channels without modification. This lets me build new drivers automatically for each alpha release, validate the drivers on the beta channel, and have no surprises on the stable channel. Does Talos intend to do something similar?
Yes we do. I personally am working on the channel based approach for our 0.3 release that we just started developing. I would love it if you could make a meeting some time soon to chat some more. User feedback will be really help.
Since Talos is built entirely in containers and we control the entire toolchain, I believe you could achieve the same with Talos.
I am your target audience and few things put me off:
- no SSH access to nodes. As you mentioned in the docs ideally we'd need node of it, but restricting me to just dmesg, ps and top via osctl won't cut it.
- due to combination of lack of SSH and custom kernel , it is probably next to impossible to run perf tool, which should be compiled using kernel headers of the currently running kernel. Same goes for bcc or any other bpf based tools
- node joins the cluster based on a static trustd username and password. It kinda defeats the purpose of all the dance around PKI if static secret is all you need to become a node. This part is hard to get right, but TPM or vTPM on cloud providers can be part of the solution
- provisioning cluster is easy part. Upgrading it is where fun begins. There should be clear demonstration that you covered this case in your fully automated OS/kubenretes installer, I couldn't find any on your website
- you introduced own components, namely trustd, osd and proxyd. There should be a diagram of what runs where and how control and data are passed between different components.
- building your own OS is a big task, you probably base yours on something, would be good to do document tha, this would make it more trustworthy
First of all, thank you for taking the time to write this out.
The feedback is very valuable.
I will do my best to address each comment.
Let me start by laying out our design constraints.
We knew we wanted a handful of simple features:
- minimal
- immutable
- and secure
and we approached them with the willingness to do whatever it took to achieve them, no matter how different it would be from any Linux distribution today.
The degree to which we want to obtain minimalism is what I like to call "ultra".
Not a single file should be on the image that isn't absolutely needed.
Furthermore, not a single process should be allowed to run that isn't required to obtain the goal of running Kubernetes.
So we started by creating an image with just enough to run the kubelet and the kubelet only.
Obviously, this isn't practical, but it was a place to start.
In implementing our immutability design constraint we decided to:
- make the root filesystem read-only
- have no package manager
- not allow any generic use of the OS (i.e. it would be only for the purposes of running Kubernetes)
When optimizing for one thing, you often degrade another.
In our case, if we optimize for minimalism, then immutability becomes degraded.
We need to address a way to manage and debug the node, and we need libraries/binaries to do so.
With no package manager, this means everything must be baked into the image, and thus we degrade immutability.
Tacking on yet another design constraint, security, things become even more interesting.
The more you add to a system, the higher the risk in vulnerabilities.
The more allowed permissions in a system, the higher the risk in vulnerabilities.
So minimalism, and immutability actually complement security.
In our case, security has the highest priority of all, which means we aren't willing to degrade anything that supports the security of the system.
So minimalism, and immutability must be present.
Aside from our design constraints of minimalism, and immutability, we also avoid C as much as possible.
We want to build something using a modern language for all the reasons you would choose a modern language over C today, but mostly for security purposes.
Taking all the above into consideration, this meant that we are still left with figuring out how to manage a machine without degrading minimalism, immutability, and security.
So without tooling on the rootfs, without a package manager, and without a way to run custom processes, we still need a way to obtain the information we need from a machine.
Thus the API was born.
The API doesn't only solve the management issue, it also reenforces all of our design constraints:
- we can keep the image minimal with a single binary serving the API
- we can keep the image immutable by building a robust API
- we could retain security by using mutual TLS and offering a read-only API
- we could write it in a modern language, using modern tooling (golang and gRPC)
At this point what need is there in SSH/console access if the design constraints essentially remove all usefulness in console access?
The problem isn't necessarily the need for SSH/console, its the need for a way to get the data to make informed decisions.
There are also additional benefits to an API.
There is a reason the concept exists.
With an API you get a standarization, strong types, and constistent and well known output formats.
The benefits are many.
I'd like to also point you in the direction of an execellent talk given this year at Blackhat: https://swagitda.com/speaking/us-19-Shortridge-Forsgren-Cont....
The section on D.I.E. in particular will add some additional support to the reasons I gave above.
That is my lengthy response to the reasoning behind the removal of SSH.
Remember, just because we don't have SSH baked in, nothing is stopping you from running a DaemonSet that has SSH.
As for a custom kernel, we would love to support this.
Happy to take in feedback here.
We create Talos in containers and our goal is to create the necessary tooling to make this dead simple.
As for node joins, they do not happen with the trustd username and password.
We use kubeadm under the hood, so its token based, and possible to have a TTL.
We have since moved to token based approach for trustd as well.
Note that the trustd token simply gives the node the ability to a worker to request a certficate for OSD, so that you can hit the node's API.
We are currently working on an upgrade operator and it is planned for v0.3.
If you would like to have some say in the direction we go, we would be happy to have you in our community meetings!
You make good points about the diagrams.
It is clear from this post that we have work to do around the documentation.
And finally, Talos is not based on any distribution.
We have a toolchain that we build, and subsequently build our entire distribution from.
I hope I have answered your questions well enough.
I look forward to hearing back from you.
Your input is valued, and we really would like to use it to turn this into somethi great!
I see immutable, but also upgradable? Is that via in-place upgrades or do upgrades require a reboot?
Example: severe bug or vulnerability in kubelet or containerd/docker. Can I use the API to roll out a fix to existing nodes such that running workloads have no disruption?
We are taking two approaches to this. The first is that you could roll out a replacement node and shutdown the old one. In bare metal scenarios this is much harder so we implemented in place upgrades, but they work very similar to creating a new node. Since Talos is immutable and runs from RAM, an in place upgrade consists of shutting down all services, and then wiping the disk and performing a fresh install. We then reboot the node and its as if you wiped the machine clean and installed the new version of Talos from the get go. This is all via the API by the way.
The rootfs in stored in the booloader partition and in the initramfs. As for NFS, I can see us adding support for that, but the out of the box experience for Talos in any of clouds will be painful if we exclusively require NFS.
I agree with this to an extent. There are certainly places where replacing can be expensive. For example, bare metal, or if the machine contains a large amount of data and moving that data to a new node is time consuming.
Your storage should be separated from the worker nodes. Unless you have some hyper-converged setup, then you make the deliberate choice that your node became special. (sorry for using the term hyper-converged)
Looks great: I run a compute cluster (30 bare metal nodes) and I can barely keep the OS patched. I've been looking to switch to a k8s setup, running out of a RO NFS root (booted with PXE). This looks like an even more promising choice.
I'm a fan of the bare metal providers, btw, though I mainly tried out Scaleway for a while.
> PXE boot Talos on every boot
IMO this is a great thing, since once your combination of tftp/dnsmasq/matchbox/http is configured and deployed, you don't need anything else.
In the current, old setup I use, I just flip the PXE-boot switch (e.g. w/ ipmitool) to image the node's drive with whatever install process and then switch back to local boot, but switching boot device and the local hard drive are both extra failure modes (at least in my aging bare metal cluster)
We do support storage volumes. A recent change in Rook seems to have broken how it works with Talos, but we know storage is important and we are working on fixes. We would love to land support for Nvidia GPU containers. Your not the first to ask for GPU support, so I'm certain we will be taking a closer look at that.
I spent a lot of time in v0.2 building Talos for ARM, and it works, but there is a good amount of work to be done to make it official. We need to setup ARM nodes to run our builds from, and refactor our build logic to account for multiple architectures. There is also a bit of work to be done around the boot loader logic since we use syslinux. We are close, just needs a little push.
while an 'os for kubernetes' would be interesting, the documentation doesn't seem to really explain how:
"Talos lets you treat the cluster as the machine, so you can focus on your applications instead of managing the OS."
all the examples seem to basically be kubectl, etc. commands with different syntax. How exactly is this an 'operating system' and not 'yet another kubernetes build / deployment utility'?
Maybe it's there. I don't see it from a cursory glance (read: clicked 10-20 pages) of the docs. Or, curmudgeonly old me is expecting the term 'operating system' to mean something else? idk. Happy to be corrected.
It is basically a Linux distribution which only supports running a single app (k8s). It has a custom init process which only supports running k8s-related processes, with no support for directly running any other services.
It is an "operating system" in the same sense that any other user space combined with the Linux kernel constitutes an operating system, so in the same sense that Android, Debian, RHEL, Ubuntu, etc., are "operating systems".
- we go to great lengths to make the OS secure and immutable
- we have an API
- we will have automated upgrades
We can allow those who are operating clusters to care far less about the OS. Managing SSH, packages, auditing requirements, etc. at the host layer is a job in itself. We aim to remove that concern and allow you to focus on Kubernetes.
What if it's mainly but not only k8s that you want installed - wireguard for inter-node networking for example, or something to support GPUs/other hardware?
We're working through a plugin system that might allow for things like this. GPUs/other hardware will obviously not be pluggable, but we have had requests for both and we are interested in adding support for them. We have really powerful tooling that makes building kernels with modifications simple.
If there are any folks here from Talos, I think there's a lot of confusion about what it is, where it fits in the stack and so on.
I think part of the confusion comes from the scarcity of diagrams. I personally also find it helpful to have design motivations laid out. There is a list of "capabilities and benefits", but it would help me to understand what the current state of the world before Talos is added to it.
Things that are called Talos from a basic web search: 1) Cisco's security group, 2) an energy company, 3) an Elder Scrolls character, 4) a cryptocurrency, 5) a high school website, 6) some sort of data science thing, 7) body armor manufacturer, 8) a recurring Marvel Universe character, 9) a biochem program, 10) a secure IBM workstation, 11) a beer dispenser, 12) a Swiss management consulting firm, ...
Sounds neat, but is it really needed? K8s doesn't really care about what it is running on. Not sure what problems it would solve. And might create others (compliance?)
Last I checked k8s required you to have swap disabled and some cgroups settings configured. As I recall, installing Kubernetes was quite difficult, and having something that's ready to go sounds great.
Installing Kubernetes has gotten easier since the time you last looked at it then. Having used Kubeadm and then RKE from Rancher, I can get a 3 node (non-HA) cluster up in under 10 minutes. This is on regular RHEL 7.4 machines when I last did this in a non-automated way.
I build and maintain large-scale Kubernetes infrastructure for a living the way I put it is: easy to setup, somewhat more challenging to maintain.
It's not rocket science, but like any complex computing tool it requires dedicated attention especially if you are going to run more than a handful of clusters. A lot of the tooling in the ecosystem falls flat here, it all solves the Day One: Getting Started problem but often punts on Day 2: Operations and then once it realizes it is actually a problem hamfistedly engineers a bolt-on solution.
So back to my point: Provisioning and setup easy... maintenance moderately more complex.
Agree 100%. Part of the value add we feel we bring is going to be an OS that keeps pace with Kubernetes. Also, automated upgrades. Making the maintenance of a cluster a little easier over time.
At it's simplest, just a couple of flags (--upload-certs from the init node, and --control-plane from the joining masters).
It gets a bit more complicated when you automate it and support adding/replacing nodes - the uploaded certs are valid only for 2h for example. (IMO the easiest solution to that is manual cert distribution - i.e. just scping them to the joining nodes.)
Security wise sounds a bit light.
I like the idea but I'd be more comfortable with a more security-first approach instead of "we are infra people" as we know how that usually ends.
I'll be following the project closely as I think the idea is good.
Can I put a pod back inside, with SSH, and use it to kubectl exec my way into an interactive state?
I get why the "outside" has no SSH, but I would not preclude the existence of a shell inside, busybox, you're back in the land of the UNIX living and all that entails.
You absolutely could. The rootfs is read only so what you can do is limited. Kubernetes also landed support for ephemeral containers in 1.16: https://github.com/kubernetes/enhancements/issues/277. It still needs more work but the idea is that you will be able to attach a debug container (you image of choice) to a pod on the fly.
The big difference seems to be removing SSH. I understand the theoretical rationale, but in practice it seems like this would complicate troubleshooting quite a bit. Yes, maybe you shouldn't be SSHing into prod ever, but how do you keep the environment consistent if you do want to allow SSH in dev/test?
We have added an API to help with the practical issues in removing SSH. In doing that it has also opened up interesting opportunities in automation that we are currently fleshing out.
The difference here is that Talos is purpose-built for Kubernetes. What that means is that we will pour resources into automated upgrades paths for our users. Tighter integration with Kubernetes, where we envision a self-healing system that makes use of the Kubernetes and Talos APIs to make decisions.
Somehow When I read "OS for Kubernetes" I thought it was yet one more layer of abstraction on top of Kubernetes.