Alternatively, you can use tools like Kubernetes and Terraform as abstractions around cloud environments while still being able to utilize them fairly effectively. Kubernetes is a clever product for Google to release, because it makes migrating loads to GCP much easier if you already run them in Kubernetes.
Similar decision for me though. If you want k8s, use Google or Azure, where they take care of it for you.
K8s on AWS, at least right now, means you are in the business of being a plumber. For example, if I remember right, the solid k8s ingress controller means ELB, which has issues like the need for cache warming. ALB is better, but has only bleeding edge support as an ingress controller in K8s. Similar for getting it installed on AWS in the first place.
Rackspace or Digital Ocean or similar is a cheaper better place if you want to roll your own k8s solution.
I'm hosting K8S on AWS in production. Kops takes care of most of the details, though I _do_ understand the low level details of what's going on. Gotta be able to debug somehow.
Google Cloud Platform is definitely cheaper than AWS, and their hosting of Kubernetes seals the deal for me. The only problem? Legacy. If I want to link two virtual machines across cloud environments, it introduces somewhat unacceptable latency, and throughput will be less reliable. So connecting to things like extremely busy Redis or AMQP clusters is not really as safe and downtime is more likely.
Still, I don't feel like Kubernetes is terrible on AWS. It works fairly well. I'm not using Ingress resources right now, just a bunch of services with their own ELBs.
Sure, I concede it can run well. It just feels like renting a furnished house and storing the supplied furniture in the basement so you can use your own.
Why not just rent a cheaper unfurnished home if you insist on bringing your own...
I'm not saying I entirely disagree. I'm mostly suggesting that moving toward systems like Kubernetes and Terraform help you to reduce lock-in so that you can pick the best tool for the job. 7 years ago, it made sense to use AWS and not look back. But I'm sitting here now, and a lot of software in our stack is pretty tied to AWS when I'd much rather use GCP.
The furniture analogy is a bit flawed. I can get comfortable in a new, furnished house, but furnishings don't come with vendor lock-in. Cloud services naturally do, at least the highly proprietary ones. I'm not saying put the furniture in the basement, I'm just saying throw a slipcover over it so we don't touch it directly. :)
DigitalOcean meanwhile is a lot more limited. The new stuff they've added is nice with firewalls and load balancers, but the tooling with AWS is more complete and I can utilize a fair bit of that tooling from within Kubernetes, including things like Amazon's certificate provisioning.
Basically to be clear, I'm saying maybe now embracing GCP and Azure seems like a solid plan, but years down the road you might want to have some more mobility. If you're already using things you can bring with you to the next provider, you're going to be ahead of the curve.
