Since the GPU libraries are hosted on the node, privileged flag is typically required to make that possible. I'm sure there will be improvements to not require privileged, but today it's mostly a requirement to get anything useful out of containers tapping into GPU.
That said, if you set the allow-privileged flag to false GPU drivers will still be installed but you may not be able to make use of the cuda cores
That's weird, because all the times I tried the experimental support, it didn't need privileged containers. From the YAML files, it looks like it's using hostPath directories, but those don't require special privileges, unless you need to write to them:
You don't need to mount the /dev entries into the container at all. The experimental support creates them automatically for you when you are using GPU resources. Perhaps it's device nodes, not the libraries that required privileges?
Aaah that is interesting. Let me dive into this later today and test my charts without that. It would actually make my life way easier for charting.
I got that from a very early stage work and never questioned it again (the /dev stuff). Thanks for pointing that out.
Gluster is a good alternative, Ceph can be prickly if you don't want a block device and instead just need a filesystem. CephFS fills sort of the same role, but does so on top of Ceph rbd.
Since GlusterFS /can use/ NFSv4 as a client, it should work with the stuff @samco_23 uses
I agree, I usually update defaults so that python => python3. However, PEP has some guidance on how this should be handled[1].
I imagine Ubuntu (Debian, et al) is following these guidelines. It would be cool to have a push for those who still depend not only on Python2 but on `/usr/bin/python` being py2 to update their app - or at least update their packaging :)
OpenShift is an interesting product, because it's not just Kubernetes, but a PAAS on top of Kubernetes packaged as a single solution. As a result you're not using k8s, but OpenShift / Origin. This has pros and cons which mainly boild down to simplicity vs lock in.
Honestly, I'd suggest you shop around, comparing CDK to OpenShift would be kind of like comparing virtual machines to a cloud provider. In that, virtual machines are implemented in a ton of ways, but cloud providers are a platform which abstracts that whole VM layer and provides a product on top of VMs.
Much in the same way we're packaging up and distributing that underlying engine. There's a lot of flexibility with that. You can use Kuberentes directly, you can leverage Helm which is what a lot of the community seems to be moving towards for package management on k8s, and then things like Deis and others implement that PAAS layer similar to OpenShift.
Why I'd recommend this method over OpenShift is the flexibility we afford you. For starters, things like Helm and Deis don't come as a part of CDK but will work out of the box on it. Since it's just delivering Kubernetes the platform. As such, if you try Deis and don't like it, you could try Kel or any other PAAS / tool built to work against vanilla Kubernetes. With OpenShift - you use it and that's the choice you're saddled with.
At the end of the day, take some time to try both. Deis is quite polished and would give you an idea of what to expect from other PAAS solutions. It's built on Helm which is part of Kubernetes so it's less deltas from what upstream is doing.
We're still "beta" so we have been able to release within 10 after upstream.
With recent updates to our build/ci process, we're able to constantly test master. This means we can bless releases days after being cut. Expect GA for CDK around 1.6.0 release time, which we will be driving for release blessed within five business days of upstream.
We don't begin integration testing at day 0 of release, we are constantly integration testing.
That said, five days is a goal of ours, not a hard deadline or SLA. An example of this was the 1.5.2. This came out 8 or 9 days after upstream because we had failing integration and e2e tests starting a few days before release and it took some time to hunt down, address, and make sure it was an issue with how we're deploying k8s and not k8s itself.
Hopefully that clarifies how we're able to to stride towards that goal. To be honest, testing like this is as much and art as it is a science and it's taken quite a while to get to where we are now (and admittedly, still a ways to go until we're satisfied).
It also said "With recent updates to our build/ci process" but there is more to testing than continuous integration, especially for something as complex as Kubernetes.
In particular, continuous integration can run regression tests but how many bugs will it find due to interactions between features?
Non-rhetorical question: Why would I use this instead of the many other approaches to running Kubernetes? Looking through the linked pages gave me no clue what the advantage is. I love Ubuntu, I've been a heavy Ubuntu user (on servers, laptops, etc.) for over 13 years, and I have a 25-node Kubernetes cluster running as part of https://cloud.sagemath.com. The kubernetes nodes all run Debian images, since that's what actually worked by default on GCE (with Kubernetes 1.3) when I setup the cluster. Soon I'll be making a new Kubernetes 1.5.x based cluster, and expanding how it's used.
Anyway, I feel like the documentation I was easily able to find about CDK misses the "comparison to the competition". I will maybe consider CDK as one option, instead of just using the install scripts that come with the Kubernetes source distribution.
First off, sage math looks awesome \m/. This is a great question, I'll be making sure we update our docs and pages to highlight these differences, here's a few that might be helpful for you:
From an installation standpoint you get the same Kubernetes /everywhere/. You can run this on all the public clouds, e.g. AWS, GCE, Azure, Rackspace, etc., and on private infrastructure like OpenStack, VMWare, and bare metal. And any Ubuntu machine, like a laptop, single server, etc.
CDK as a whole is really just applying operational knowledge around Kubernetes. We are not adding any additional bits to k8s, we distribute the same binaries as upstream. If you stop using our tools, you can still manage the cluster as if you'd stood it up by hand. Since we're distilling operations, and not just installation, we cover the entire breadth of lifecycle tasks. Come 1.5.3, 1.6.0 (+ etcd3), and beyond we work to make sure tasks like upgrades work reliably and consistently.
We also present a consistent interface to common maintenance tasks, so that everyone using the solution uses the same primitives for maintenance. This helps eliminate the need to SSH and hunt around for these tasks. All the operational code we produce is open source, it lives in the upstream tree, and we love contributions!
Our roadmap for features is driven almost exclusively by our users, both community and commercial.
I could probably write pages, but I think those are probably the starting points for things we're doing as a distribution of Kubernetes. If you have any specific issues while running k8s, I'm happy to help answer how we handle those, if we do.
How is the dynamic adding / removing of nodes story? I am really waiting for something like kubeadm to become stable, and be able to just run "kubeadm join X.X.X.X --token XYZ" and be good!
It's super clean already with CDK. You just issue a single command from your laptop, something like: `juju add-unit k8s-node` or `juju add-unit -n 10 k8s-node`. This will provision a new machine wherever you're running Kubernetes (AWS/GCE/Azure/Rackspace/OpenStack/VMWare/Bare metal/locally), install kubelet, configure the SDN you've chosen, and have it available to schedule workloads.
As kubectl moves to ready status, we'll look to use it more under the covers to help conform to upstream's recommendations.
Looking to install on DreamCompute, which supports OpenStack APIs. I just installed it, but don't see an option to install it on OpenStack. Is deployment via the OpenStack API supported? The linked article does say "designed to be easily deployable to public clouds, on-premises (ie vsphere, openstack)"
Thanks! I got through all of those steps but then hit the same error as you. I think this could be the standard for deploying K8S on DreamCompute once it works!
Unlike other companies, we don't jump in to upstream projects and throw weight around to influence a project to make it more marketable for us. Kubernetes has a vibrant, and powerful core contributor base already. When it makes sense, for any changes we make, we'll be happy to package those back upstream. Primarily, our contributions to Kubernetes is around operations. What happens 1 week, 1 month, 1 year after setting up a cluster.
GKE is great, but GKE is Google only. It's not on-prem, it's not cross cloud, and it's not portable. That's important to some people. Our contributions to cluster/juju is the distillation of our operational knowledge in running Kubernetes everywhere. The same upstream k8s, deployed with the same tooling, everywhere.
The way you start this off is a pretty poor response to my original question, whereas the rest is fantastic. It isn't about throwing one's weight around or even making it more marketable, but about improving the product for new use cases (such as Apprenda wanting to better Kubernetes via the on-premise sig).
Thanks for your response, I was curious as to the value add and this helps.
Thanks for the feedback, my original sentence may have been a bit hasty, but the core of the message is there. I see Canonical's role as the expediter/server in a kitchen instead of donning another chef hat when dish are piling up to go out. We want to celebrate and get the amazing work of the community into as many hands as possible.
We participate in SIGs as well, sig-on-premise being one we co-chair and co-founded. We've planning on helping the project in ways other than code contributions.
Sorry I meant even Apprenda has actual golang feature commits. That was my point Re: Apprenda and contributions.
This seems to be Canonical's general direction. It just makes me sad that is is more on marketing than engineering. I wish it was more of both as clearly Canonical does Linux + marketing better than basically anyone or there wouldn't be so much Ubuntu everywhere :)
That said, if you set the allow-privileged flag to false GPU drivers will still be installed but you may not be able to make use of the cuda cores