There was a very interesting talk at Google Next in March that described a way in which the Kubernetes open source team is bringing this sidecar architecture to Kub. In fact I think they are working with the Lyft team. The whole talk is interesting but I've added a time offset to skip to the bit where they talk about sidecar/service mesh.
This is a battle that repeats itself ever so often. Same deal with kubernetes config files.
It's the same opinionated positions of puppet vs chef/ansible. Building the tooling in the form of an embedded programming language (lua for nginx, python for ansible) is so much better.
But that is a much more polarising position to take. It's politically untenable. Ruby engineers will say no to python and vice versa... So you start arriving at Json or XML based configurations which are a programmatic neutral ground.
IMHO lua-jit occupies the same political neutrality as XML.
I remember everyone complaining about Spring and similar having these massive XML configuration files, and then people said "never again!", and moved to very simple JSON formats for their frameworks. And then everyone adds a little bit to the best practices, and over time we get this, again.
1. quite hard to "give it a spin" without using docker (vomit). I eventually found some random github repo that had centos7 compatible build scripts, and it took quite a while to compile it on a test vm since it had to build gcc and a ton of other stuff. Not sure why they can't just supply a static binary "release".
2. The config really is a mess. I managed to get something almost working (only / requests went through the proxy, but nothing else for some reason, even though I was using a prefix config). There aren't really any good example configs I could find either. There is a weird config builder script with json templates in the configs dir, but it just seemed to spew out some preconfigured madness.
I am sure it works great and has more tunables than you can shake a stick at, but for now it seems to be only usable by people who have lots of time to invest in trying it out.
That's how they use it, that's not how you have to choose to use it. It is one of the most tedious aspects of the tool but all the different aspects exist for a reason, there just needs to be an easy mode for people starting off.
The example you linked isn't so bad. Perhaps a bit verbose for a simple example. The formatting of the json is ugly though, and I think they would do better to not collapse the whitespace inside arrays.
Seems like they could have used something a bit nicer to work with too (yaml, libucl, hocon, etc). I am honestly getting tired of editing fiddly json (no comments, last element in list can't have a comma, etc) all over the place.
You get big files filled with variables, loaded from other big files also full of variables. It's hard to know what you are setting after a while or where it comes from.
After a while playing with configuration systems, I'm starting to think that configuration should just be hardcoded once and for all.
Envoy is an extremely exciting project. I've been following it closely since OSS inception and am thrilled to see the upcoming deep integration with the Kubernetes project. Such a natural and complimentary fit!
With flexibility as both a side car and host proxy network call fabric for polyglot distributed services, I see Envoy as a kind of modern ESB for the reincarnation of SOA we are seeing in the microservices movement. Congrats to the awesome contributors from Lyft/IBM/Google and others!
Interesting that it seems like there is a bunch of overlap with linkerd in the k8s space. With both k8s and linkerd being CNCF projects, I wonder why google is putting so many resources behind envoy rather than linkerd. I guess that's the beauty of the k8s ecosystem though, freedom and choice to use what you see fit. K8s just provides the rock-solid primitives.
One downside to using Linkerd is that unless you have very few apps — or an overabundance of RAM — you can't realistically deploy it as a "sidecar container". The JVM consumes about 100MB of RAM, by their account; it's not lightweight.
The sidecar pattern (used by Envoy and by Google's new project, istio) is desirable in that it compartmentalizes the rules and logic in the pod that uses them, as opposed to sharing all settings across all apps. Zero dependencies (between pods) is a great way to manage complexity. I haven't thought much about it, but there might be latency benefits, too.
This. Linkerd has been a pretty cool piece of tech. In fact, their rewrite linkerd-tcp is in Rust and is blazingly fast. Also, it works on L4 and can do more stuff than Envoy... though I think that advantage is going to be shortlived.
Linkerd guys have been focusing on building deep integration with k8s (as an ingress or sidecar), but this announcement of Envoy trumps it all.
>"We are excited to announce that we are working in partnership with both Google and IBM to bring Envoy to Kubernetes. Fun fact: there are now more people working on Envoy at Google than there are at Lyft! We have a lot of other things planned with Google that we will be able to share more about in the coming months."
That's a bummer for the amazing guys at Buoyant. I had hoped for linkerd to be the next generation of ingress in k8s.
Envoy is in C++, while linkerd-tcp is in Rust. I wonder how much role did the choice of tooling play in Google's decision to adopt.
We've been experimenting with Envoy for a few months and have been really impressed with the technology and the responsiveness of the community. We're actually deploying an API Gateway built on Envoy so you can easily extend your service mesh all the way out to the edge. Planning on open sourcing it soon (if anyone wants to give it a try, my email is in my profile).
Sorry, I wasn't clear. I was referring to using Envoy as an API gateway, not a specific API gateway product. We had to write some wrapper code on top of Envoy that lets you use a REST API to map URLs to services running in your Kube cluster and then dynamically update Envoy config, etc.
Envoy bills itself primarily as a service mesh (seeming closest to GSLB / Global Software Load Balancer), but also as an edge proxy (seeming closest to GFE / Google Front End).
Is there any reason to think that it's more like GFE than it is like GSLB?
Thanks.. just tried it out with my service, beats nginx for simple http proxying pretty easily by a nice margin. Envoy supports way more features than nginx too !
https://www.youtube.com/watch?v=3quCoi5YHz4#t=27m59s
Also, there is https://istio.io/ but it's a placeholder site at the moment.