1. We are considering Istio! This is especially true for our Kubernetes environment. We are already planning to deploy Pilot for the first iteration of our control plane in our non-K8s environment, so the other pieces that comprise Istio is a natural place for us to continue exploring.

2. We have not containerized prior to Envoy. We're running application binaries provisioned with Puppet on EC2 for most of our infrastructure still.

2. Do you have multiple non-containerized services per machine? If yes, did you go with one proxy per machine or service instance?

We run one proxy per machine, even when there are multiple services running. The proxy is just an abstraction to the downstream dependencies. Even if there are multiple services per machine, they can still reach downstream services via the same proxy path.

Thanks for all the answers, I really appreciate it! I've got one more question.

In the period you had parts of your system with envoy and parts without, have your routed the outbound traffic from envoy-equiped services through their local proxy before reaching its envoy-less destination? Or did you omit envoy then?

We route all outbound traffic from internal services through Envoy, even if the destination isn't running Envoy. We don't have envoy running as a "front" proxy right now, i.e. our L4 setup isn't Envoy <-> Envoy, it's Envoy -> service directly. An example of this is the DB layer - traffic going to our DBs from services goes through Envoy service-side but Envoy isn't running on our DB instances.

What are the reasons for your to have chosen to do so? I mean going for a "back" proxy instead of envoy-envoy (which seems to be the most "advertised" approach) or "front" proxy. As I understand, this way you're losing the envoy features for your most "shallow" service. Or do you also run envoy on your ingresses?

The "back" proxy was the initial setup with SmartStack, so we went with that for minimal viable first steps. We wanted to make incremental changes, changing as little as possible, for this migration so we could monitor for correctness and performance at every step. The eventual plan is to run Envoy as a front proxy for ingress, and maybe even Envoy <-> Envoy everywhere, where we have Envoy as both a front and back proxy on every service deployment (instance, container, etc.)

Ok, thanks!