So the war for enterprise on-prem Kubernetes just intensified :) RedHat vs Google? :] I wonder how is Rancher doing? I guess life will be harder too?
Timing seems to be important, just before OpenShift finishes absorbing Tectonic (or maybe Tectonic is absorbing OpenShift) with their installer, UI and billing services. I guess there's still time for Google to land some big deals :)
OpenShift is the most mature Enterprise Kubernetes that can be deployed and managed in any cloud environment (public or private). It's great to see further validation that customers want to run applications in both existing data center environments and public cloud environments.
Will Container-Optimized OS be used as the operating system for on-prem? If so, any plans to spin this off as a more general purpose OS now that it needs to support on-prem use cases?
Hi Weston, I'm exploring and using Istio in the recent months, so what is the advantage of the "Managed Istio(with commercial level support)" over the community version? As I know the community is still actively optimizing the performance, will the Managed Istio have a better performance?
> With GKE On-Prem, you get the Google Kubernetes Engine (GKE) experience directly in your data center. A quick and simple install and upgrade experience that’s validated and tested by Google. GKE On-Prem also registers your cluster with Google Cloud Console in order to have a single-pane-of-glass view for managing all your clusters.
That latter is an interesting way to mentally merge your local DC and Google Cloud.
It will get even more interesting if you get decent federation support between your local GKE clusters and GCP GKE cluster allowing you to scale out to the cloud if you run out of capacity in your local cluster.
Exactly. This is really the beginning of multi-cluster scenarios that work well across different environments. Failover from on-prem -> GKE is something we're working on.
We're going to give a breakout session that goes into more depth on Wednesday @ 4:35pm. IO244.
Some quick details: It's a bit of a split between what GKE runs and what the customer runs. Alpha runs on vSphere 6.5 and we're packing up a Google-hardened OS in much the same way we package GKE for GCP. A lot of the integrations for things like networking and storage will be coming from partners. We'll also have remote mgmt capabilities so we can manage the cluster's control plane in much the same way our SREs do for GKE.
Will this be something like COS or even CoreOS? Also, I'm curious to hear more about this part:
> GKE On-Prem has a fully integrated stack of hardened components, including OS, container runtime, Kubernetes, and the cloud to which it connects.
Which runtime are you shipping? CRI-O? What type of outgoing cloud connection is that? I have so many questions. I'm actually at the conference this week if you're willing to grab coffee.
Exactly. As described, it sounds a bit like magic. Of course, it's not. Thus, one wonders about what happens in the lower layers of the stack, namely who gets to run them (GKE or the HW owners):
* DNS (not the Kubernetes in-cluster one like kube-dns)
* DHCP
* LDAP or equivalent
* SSH and its keys
* on-prem security of the cloud identities (does it require TPM? SGX?)
Not to mention probably the hardest bit, which is how does it do persistent storage? Running k8s on the various cloud providers tends to use storage engines for those providers (ebs, etc)...
Does it ship with ceph out of the box? Some in-house block store? What happens when it breaks? Persistent volumes are IMO the very hardest thing to get right, and for me it's the big reason why I'd rather put my trust in a hosted solution in the first place.
If they solve this, and make it as seamless and easy as using cloud storage offerings, they've completely changed the game. Somehow I think they have a ways to go.
I can attest that persistent storage is the hard part! Full disclosure, I work for a company[1] who makes a persistent storage solution for containers/Kubernetes. We are absolutely seeing that our large customers (folks like GE, Verizon, Dreamworks, Comcast, etc) are running "cloud native" applications on-prem as well as in the public cloud so this is a really smart move for Google.
I assumed they would punt persistent storage for the first release, kinda like they launched Cloud Filestore with only NFSv3 support and without snapshots. As you say, it's a tough problem to tackle. If they to ship with something, I'd expect them to go for Ceph. As far as I can tell, if you squint hard enough, its lowest layers are the ones that resemble the most those at Google (Colossus/D).
Do you have a source for this? Is there documentation anywhere that says GKE On-Prem is using rook? Or are you just saying "people who use kubernetes on-prem often use rook.io"?
Jesse here. I'm the eng manager for GKE and GKE On-Prem Storage lifecycle over at Google. For basic persistent volumes, we have working vSphere (block) support, which allows us to do a lot (including persistent/stateful services).
We also have great storage abstraction layers built into K8S - CSI, FlexVolumes, and a large suite of in-tree plugins - so adding additional ones is pretty easy.
I think you're talking about something else however - specifically scaled/distributed storage services.
We're investigating options here, though keep in mind you should have no problems running containerized storage services in GKE On-prem. They would be on top of the existing block support I mentioned above. I saw a couple comments in other threads from some vendors that sell solutions that do just that.
For storage systems that don't containerize, that is a different discussion. Happy to talk more.
Is this ever going to be a bare-metal thing? Like probably many others, I'm not really interested in doing on-prem virtualization... kubernetes is interesting to me because containers are a better abstraction than virtual machines in the first place. Why add a virtualization layer if you don't have to?
(I get that it makes your life easier as the developer of this product, but having to run a virtualization IaaS between your metal and your orchestration makes the whole thing rather uninteresting IMO.)
I feel like I have to say this, even if people get it already.
Walk before you run.
Bare metal is a LOT harder to manage because, well, hardware fails. We hear the demand, for sure, but vSphere represents walking (and has a lot of customers, too :)
But, I don't buy the hardware failure argument, because the same is true of running a vsphere installation in the first place.
vsphere migrates VMs to other machines when the hardware fails, but, analogously, the kube scheduler moves pods to other machines when they fail as well. You have to worry about disk failures in both cases. You have to worry about keeping your vsphere's database up and in a high-availability mode (postgres in my experience), just as you have to worry about keeping k8s's etcd cluster up and in a high-availability mode.
For any problem k8s has on bare metal due to hardware unreliability, vsphere has an analogous problem, it's just pushed down one layer.
IMO the real reason why this is a pragmatic decision is because, people already have lots of experience in running vsphere and understand where the risks and challenges are, and vsphere has lots of tools for things like automating the installation of the hypervisor OS, base level network setup, expectations around NFS for VM storage, etc.
Vsphere represents a decent, known set of tools for getting an infrastructure up and running on bare metal, which is a prerequisite for getting kubernetes running, but what would be exciting to me would be a rethinking of those infrastructure components in a purely open source and industry standard fashion, in a no-frills way that only exists to get a basic k8s control plane up.
so what is the model for bare metal with GKE on-prem. The reason is for VSPHERE 6.5 is additional cost (license per cores to VMWARE and VCENTER license) which we want to avoid to use bare metal only.
I previously worked on a similar (non-Google) product that was similar -- it was an "On-Premise Cloud" [0], where the cloud provider managed and owned all the hardware and software, and the customer created workloads on it, and the physical hardware was scaled up/down based on demand.
The product worked well, but I think there was an uphill battle in explaining the mechanics of the arrangement to customers.
Masters will run on-prem. We have connection agent that let's us securely talk to the Kube API Server from GCP. We wanted to ensure that the cluster is fully functional even if the connection goes down.
Great to see GKE coming to enterprise datacenters! IBM has been very successful with IBM Cloud Private (https://github.com/IBM/deploy-ibm-cloud-private) bringing an enterprise Kube distribution for VMWare/OpenStack/Bare Metal in enterprise datacenters since last year. I love to see the momentum of another Kubernetes distribution helping create the de-facto next generation of apps for all kinds of use cases.
We have an in-house cloud running Kubernetes and we also use Google Cloud with Kubernetes. It is scary to move to GKE in-prem if we do not know pricing.
While the cluster can operate in a disconnected state, much of the functionality is provided by the connection to GCP. Things like UI integration, policy syncing, Stackdriver, etc. Our early focus is on datacenters that have a connection to the internet. However, we're starting to look a lot more are airgapped environments.
GKE On-Prem is packaged with upstream K8s. So for your team that currently uses `kubectl` to deploy or manage workloads, there won't be any differences.
GKE On-Prem is a Google provided, validated and supported distribution of Kubernetes and extensions that offer a GKE-like experience in your on-premise datacenter. It makes it easy to install and upgrade Kubernetes and provides access to GCP services such as monitoring, logging, metrics, security and auditing for your on-premise installation. It is the foundational component of the Cloud Services Platform, and is how Google "brings the cloud to you".
CSP combines Kubernetes both in your on-premise datacenter (GKE On-Prem) and Google-managed Kubernetes in GCP (GKE) with Istio and other CI/CD (Cloud Build) and serverless (Knative) products. You can leverage this suite of products to both modernize your existing on-premise applications and build new applications in the cloud.
Additionally, Google will be offering phone and email support similar to the existing GCP support packages.
What is the benefit, if you get the same output from "kubectl" as regular GKE, or from any other distro.
Basically, this is yet another paid packaged Kubernetes distribution, that has the explicit goal to do "Hybrid clustering" so that it is easier to lure the customer back to GKE. Do I get that right ?
What we have found out is that most on-prem customers are eager to move to the cloud. Practically it's not easy to just lift-and-shift. So think of this is a ramp to the cloud.
Now, the benefit of upstream K8s is that your dev team can build apps and containers without proprietary APIs; so when you are ready to move to the cloud you are not locked-in.
Thanks. I agree that lift and shift never happens easily in real life.
That being said, why would I not use the actual free upstream Kubernetes for my on-prem distribution ?
(with the help of one of the thousands installer out there like kube-adm, kubespray, etc).
What I have seen working with Kubernetes for quite a while, is that the lowest common demominator is the YAML definitions for your workloads (what you want to run on your Kubernetes cluster). Those should be portable accross any Kubernetes distribution, on-prem or on the cloud. As far as I can tell, today this is already the case.
Is the benefit in this case that you can use the Google ecosystem for logs etc ?
> That being said, why would I not use the actual free upstream Kubernetes for my on-prem distribution ? (with the help of one of the thousands installer out there like kube-adm, kubespray, etc).
None of them actually provision your infra for you (VMs, LB rules etc). GKE On-Prem will.
Ok, thanks I got it, you are bundling everything into a VMWare image, that boots ready to use.
Is it fair to say that this is similar to Canonical Ubuntu MAAS + Juju Kubernetes? I'm sure that Red Hat Openshift must have something similar also to install directly on a pool of managed bare metals.
The entire cluster is on-prem. At the moment, you can optionally leverage a secured tether to manage your cluster in GCP with the same management features you've come to expect with GKE proper. If the connection is lost, your cluster is still fully functional, so no there is no requirement for permanent access to your intranet. The access, when it exists, is also secured to only permit specific access between Google's network and your cluster.
Yes and no. The connection is needed to show your workloads and other cluster information in GCP Console (similar to what was shown in the keynote). However, if the connection goes down, your cluster will not stop working.
Is vSphere Essentials Plus supported as we have a good use case. Are there any specific additional requirements eg. vsan or does it all run on the base Esxi product?
there are literally 60+ startup distributions that try to sell a "distribution as a service" for the upstream code that is free.
I think most if not all of them will fail, and as usual, the big 3 or 4 will win the market (if I had to bet: Google, Red hat, Canonical and maybe the guys at Heptio that are really cool and got the right attitude)
Given the massive shambles eks turned out to be, that would be great.