Wireguard is a really nice clean transport protocol, but there are a few:

(1) We use AES for performance and other reasons. ChaCha is much faster than AES in software but AES blows it away by 2-4X in hardware. We have customers using ZeroTier in an SD-WAN setting who are leveraging AES acceleration (with some of their own custom code) in network accelerators and other devices. We also think we can beat both in-kernel WG and ipsec in V2 on Linux using io_uring because the cryptographic part of ZeroTier is 2-4X faster than WG on "real hardware." My M1 Mac does 2GiB/sec/core and newer Intel/AMD chips with AVX VAES can do as much as 10GiB/sec/core.

(2) We have a bunch of customers who want FIPS. Yes, we know. We're not big fans either but they're large customers and whole industries. We also have a pretty good no-compromise plan for this that treats FIPS as a floor for cryptographic quality not a ceiling and doesn't add meaningful complexity. This is a V2 thing. (You can bootstrap WG with keys from a NIST ECC key exchange to get the asymmetric part of FIPS, but ChaCha is not FIPS.)

(3) We want to maintain backward compatibility with millions and millions of ZeroTier devices out there, at least as far back as we can.

(4) ZeroTier uses a construction where the identity is the outer envelope and then (in V2) the ephemeral forward secrecy keys are the inner envelope. Wireguard does it the other way around. Both are equivalent in terms of cryptographic strength (and I think in noise this is a choice, but it's been a while since I read through that). The advantage of the WG construction is that session participants can't be identified from packets. That's a disadvantage in enterprise though, because they want that. It lets us implement elegant solutions to making a mesh P2P network play nice with enterprise firewall and internal/DMZ isolation practices among other things. It also lets our roots/relays basically do something analogous to MPLS, pumping vast amounts of traffic with virtually no CPU overhead because they don't need to decrypt anything.

We toyed around with making an AES fork of Wireguard, basically use the exact same construction as WG with AES-GMAC-SIV instead of ChaChaPoly. Would be totally fine cryptographically but see #4.

We also support virtualization at layer 2 rather than just layer 3, which is important for a bunch of our industrial use cases. It would be possible though to stuff L2 into WG pretty easily, so that's not really a reason. WG is just a transport protocol so you could stuff anything into it. Fly.io used it as a light alternative to TLS I think. It's just normally used to carry L3 IP.

Edit: also in V2 we are modularizing things, separating the transport protocol from the rest of the code. It'll be in open source Rust and very portable. In our (currently private) Rust repo vl1/ is only about 5K lines of Rust including tons of comments and non-crypto stuff. We also have plans to document it a lot better.

> ZeroTier uses a construction where the identity is the outer envelope

Does this mean that any node along the network path would be able to track communications between nodes? Maybe a snoop wouldn't be able to pin an envelope to a named identity, but they would be able to track which other identities its connecting to down to the communication bandwidth between each source and destination and correlate the identities of those connections between separate devices. Is this an accurate description?

Yes, and that's a downside. My opinion though is that unless you are tunneling every single bit of your network traffic through something that is concealing your identity, fingerprinting the vast majority of devices and users is trivially easy. There is just an amazing amount of surface area: browser fingerprinting, protocol version fingerprinting, network access pattern fingerprinting, DNS snooping, TCP fingerprinting, and so on. Since ZT (and Tailscale and Nebula and other similar things) are all P2P it's trivial to fingerprint someone by just collecting the IP endpoints they routinely communicate with. Anyone who belongs to a private net with a few fixed points can be easily identified over time by the fact that they talk to those things in a certain pattern.

I have some thoughts about features to really defeat this. We've got an entry on our feature queue for guard nodes you can run in the cloud and have also researched implementing a form of "light" onion routing. It wouldn't be intended to be as strong (or as slow) as Tor, but sufficient to prevent passive dragnet surveillance.

In any case there is a conflict here between goals. So far the performance, rapid startup, and enterprise goals have won out over the small privacy benefit of the ephemeral-outer-key construction. It's a scope thing. So far we've never scoped ZeroTier as a meta-data-concealing protocol like Tor, just as a P2P mesh protocol for easy remote access between devices and locations.

Still... it's tempting to try to find something in the V2 design to at least mitigate this issue. It's something we're thinking about.

I like the guard nodes idea ("gateway" or "proxy" nodes?). Maybe introduce a gateway node type, and add a way to pass traffic through a particular gateway (with a second tor-like encryption layer) defined using an extension to the current flow rules language. That would collapse all the traffic behind the gateway into a single identity (the gateway's), and probably be general enough to support all kinds of fancy features like onion-routing, monitored link balancing, etc.

Yeah.

I think a TL;DR on my reply above:

I think the difference in privacy between identity(ephemeral(x)) and ephemeral(identity(x)) is not significant. It's more honest to tell people that our product offers only content privacy not meta-data privacy unless we are truly implementing strong countermeasures against tracking by ISPs or others that can pervasively sniff the net. Just nixing the identity key visibility is not a strong countermeasure for the reasons I described.