To be honest I kind of find myself drifting away from gRPC/protobuf in my recent projects. I love the idea of an IDL for describing APIs and a great compiler/codegen (protoc) but there's just soo many idiosyncrasies baked into gRPC at this point that it often doesn't feel worth it IMO.
Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc.
Varlink (https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.
My favorite serde format is Msgpack since it can be dropped in for an almost one-to-one replacement of JSON. There's also CBOR which is based on MsgPack but has diverged a bit and added and a data definition language too (CDDL).
Take JSON-RPC and replace JSON with MsgPack for better handling of integer and float types. MsgPack/CBOR are easy to parse in place directly into stack objects too. It's super fast even on embedded. I've been shipping it for years in embedded projects using a Nim implementation for ESP32s (1) and later made a non-allocating version (2). It's also generally easy to convert MsgPack/CBOR to JSON for debugging, etc.
There's also an IoT focused RPC based on CBOR that's an IETF standard and a time series format (3). The RPC is used a fair bit in some projects.
What I really like about protobuf is the DDL. Really clear schema evolution rules. Ironclad types. Protobuf moves its complexity into things like default zero values, which are irritating but readily apparent. With json, it's superficially fine, but later on you discover that you need to be worrying about implementation-specific stuff like big ints getting mangled, or special parsing logic you need to set default values for string enums so that adding new values doesn't break backwards compatibility. Json-schema exists but really isn't built for these sorts of constraints, and if you try to use json-schema like protobuf, it can get pretty hairy.
Honestly, if protobuf just serialized to a strictly-specified subset of json, I'd be happy with that. I'm not in it for the fast ser/de, and something human-readable could be good. But when multiple services maintained by different teams are passing messages around, a robust schema language is a MASSIVE help. I haven't used Avro, but I assume it's similarly useful.
Software lives forever. You have to take the long view, not the "rn" view. In the long view, NFS's XDR or ASN.1 are just fine and could have been enough, if we didn't keep reinventing things.
It's mind-blowing to think XDR / ONC RPC V2 were products of the 1980s, and that sitting here nearly forty years later we are discussing the same problem space.
Probably the biggest challenge with something like XDR is it's very hard to maintain tooling around it long-term. Nobody wants to pay for forty years of continuous incremental improvement, maintenance, and modernization.
Long term this churn will hopefully slow down, it's inevitable as we collectively develop a solid set of "engineering principles" for the industry.
I'm actually super excited to see how https://www.sovereign.tech turns out long-term. Germany has a lot of issues with missing the boat on tech, but the sovereign tech fund is such a fantastic idea.
Buf seems really nice, but I'm not completely sure what's free and what's not with the Buf platform, so I'm hesitant to make it a dependency for my little open source side project ideas. I should read the docs a bit more.
Buf CLI itself is licensed under a permissive Apache 2.0 License [0]. Since Buf is a compiler, its output cannot be copyrighted (similar to proprietary or GPL licensed compilers). DISCLAIMER: I am not a lawyer.
Buf distinguishes a few types of plugins: the most important being local and remote.
Local plugins are executables installed on your own machine, and Buf places no restrictions on use of those.
Remote plugins are hosted on BSR (Buf Schema Registry) servers [1], which are rate limited. All remote plugins are also available as local plugins if you install them.
It's worth to mention that the only time I've personally hit the rate limits of remote plugins is when I misconfigured makefile dependencies to run buf on every change of my code, instead of every change of proto definitions. So, for most development purposes, even remote plugins should be fine.
Additionally, BSR also offers hosting of user proto schemas and plugins, and this is where pricing comes in [2].
> I love the idea of an IDL for describing APIs and a great compiler/codegen (protoc)
Me too. My context is that I end up using RPC-ish patterns when doing slightly out-of-the-ordinary web stuff, like websockets, iframe communications, and web workers.
In each of those situations you start with a bidirectional communication channel, but you have to build your own request-response layer if you need that. JSON-RPC is a good place to start, because the spec is basically just "agree to use `id` to match up requests and responses" and very little else of note.
I've been looking around for a "minimum viable IDL" to add to that, and I think my conclusion so far is "just write out a TypeScript file". This works when all my software is web/TypeScript anyway.
Now that's an interesting thought, I wonder if you could use a modified subset of TypeScript to create a IDL/DDL for JSON-RPC. Then compile that schema into implementations for various target languages.
Same; at my previous job for the serialisation format for our embedded devices over 2G/4G/LoRaWAN/satellite I ended up landing on MessagePack, but that was partially because the "schema"/typed deserialisation was all in the same language for both the firmware and the server (Nim, in this case) and directly shared source-to-source. That won't work for a lot of cases of course, but it was quite nice for ours!
State of the art for both gzipped json and protobufs is a few GB/s. Details matter (big strings, arrays, and binary data will push protos to 2x-10x faster in typical cases), but it's not the kind of landslide victory you'd get from a proper binary protocol. There isn't much need to feel like you're missing out.
The big problem with Gzipped JSON is that once unzipped, it's gigantic. And you have to parse everything, even if you just need a few values. Just the memory bottleneck of having to munch through a string in JSON is going to slow down your parser by a ton. In contrast, a string in Protobuf is length-encoded.
5-10x is not uncommon, and that's kissing an order of magnitude difference.
> have to parse everything, even for just a few values
That's true of protobufs as much as it is for json, except for skipping over large submessages.
> memory bottleneck
Interestingly, JSON, gzipped JSON, and protobufs are all core-bound parsing operations. The culprit is, mostly, a huge data dependency baked into the spec. You can unlock another multiplicative 10x-30x just with a better binary protocol.
> 5-10x is not uncommon
I think that's in line with what I said. You typically see 2x-10x, sometimes more (arrays of floats, when serialized using the faster of many equivalent protobuf wire encodings, are pathologically better for protos than gzipped JSON), sometimes less. They were aware of and worried about some sort of massive perf impact and choosing to avoid protos anyway for developer ergonomics, so I chimed in with some typical perf numbers. It's better (perf-wise) than writing a backend in Python, but you'll probably still be able to measure the impact in real dollars if you have 100k+ QPS.
Yeah this is something people don't seem to want to get into their heads. If all you care is minimizing transferred bytes, then gzip+JSON is actually surprisingly competitive, to the point where you probably shouldn't even bother with anything else.
Meanwhile if you care about parsing speed, there is MessagePack and CBOR.
If any form of parsing is too expensive for you, you're better off with FlatBuffers and capnproto.
Finally there is the holy grail: Use JIT compilation to generate "serialization" and "deserialization" code at runtime through schema negotiation, whenever you create a long lived connection. Since your protocol is unique for every (origin, destination) architecture+schema tuple, you can in theory write out the data in a way that the target machine can directly interpret as memory after sanity checking the pointers. This could beat JSON, MessagePack, CBOR, FlatBuffers and capnproto in a single "protocol".
And then there is protobuf/grpc, which seems to be in this weird place, where it is not particularly good at anything.
Protobufs have a massive data dependency baked into the wire format, turning parsing into an intensive core-bound problem.
Interestingly, they're not usually smaller than gzipped JSON either (the compression it has built-in is pretty rudimentary), so if you don't compress it and don't have a stellar network you might actually pay more for the total transfer+decode than gzipped JSON, despite usually being somewhat faster to parse.
The docs [0] are fairly straightforward. I'll spit out a little extra data and a few other links in case it's helpful. If this is too much or not enough text, feel free to ask followup questions.
As far as data dependencies are concerned, you simply can't parse a byte till you've parsed all the preceding bytes at the same level in a message.
A naive implementation would (a) varint decode at an offset, (b) extract the tag type and field index, (c) use that to parse the remaining data for that field, (c1) the exact point in time you recurse for submessages doesn't matter much, but you'll have to eventually, (d) skip forward the length of the field you parsed, (e) if not done then go back to (a).
You can do better, but not much better, because the varints in question are 8 bytes, requiring up to 10 bytes on the wire, meaning AVX2 SIMD shenanigans can only guarantee that you parse 3 varints at a time. That's fine and dandy, except most fields look like 2 varints followed by some binary data, so all you're really saying is that you can only parse one field at a time and still have to skip forward an unpredictable amount after a very short number of bytes/instructions.
If you have more specialized data (e.g., you predict that all field indexes are under 32 and all fields are of type "LENGTH"), then there are some tricks you can do to speed it up a bit further. Doing so adds branches to code which is already very branchy and data-dependent though, so it's pretty easy to accidentally slow down parsing in the process.
Something close to the SOTA for varint decoding (a sub-component of protobuf parsing) is here [1]. It's quite fast (5-10 GB/s), but it relies on several properties that don't actually apply to the protobuf wire format, including that their varints are far too small and they're all consecutively concatenated. The SOTA for protobuf parsing is much slower (except for the sub-portions that are straight memcopies -- giant slices of raw data are fairly efficient in protos and not in JSON).
This isn't the best resource [2], but it's one of many similar examples showing people not finding protos substantially faster in the wild, partly because their protos were bigger than their json objects (and they weren't even gzipping -- the difference there likely comes from the tag+length prefix structure being more expensive than delimiters, combined fixed-width types favoring json when the inputs are small). AFAICT, their json library isn't even simdjson (or similar), which ought to skew against protos even further if you're comparing optimal implementations.
In terms of protos being larger than gzipped json, that's just an expected result for almost all real-world data. Protobuf adds overhead to every field, byte-compresses some integers, doesn't compress anything else, and doesn't bit-compress anything. Even if your devs know not to use varint fields for data you expect to be negative any fraction of the time, know to use packed arrays, ..., the ceiling on the format (from a compression standpoint) is very low unless your data is mostly large binary blobs that you can compress before storing in the protobuf itself.
For a few other random interblags comparisons, see [3], [4]. The first finds protos 3x-6x faster (better for deserializing than serializing) compared to json. The second finds that protos compress better than json, but also that compressed json is much smaller than ordinary protos for documents more than a few hundred bytes (so to achieve the size improvements you do have to "cripple" protos by compressing them).
If you start looking at the comparisons people have done between the two, you'll find results largely consistent with what I've been saying: (1) Protos are 2x-10x faster for normal data, (2) protos are usually larger than gzipped json, (3) protos are sometimes slower than gzipped JSON, (4) when you factor in sub-par networks, the total transfer+decode time can be much worse for protos because of them being larger.
As a fun experiment, try optimizing two different programs. Both operate on 1MB of pseudo-random bytes no greater than 10. Pick any cheap operation (to prevent the compiler from optimizing the iteration away) like a rolling product mod 256, and apply that to the data. For the first program (simulating a simplified version of the protobuf wire format), treat the first byte as a length and the next "length" bytes as data, iterating till you're done. For the second, treat all bytes as data. Using a system's language on any modern CPU, you'll be hard-pressed to get an optimized version of the length-prefixed code even as fast as 10x slower than an un-optimized version of the raw data experiment.
Cap'n proto and flatbuffers (whether gzipped or not), as examples, are usually much faster than both JSON and protobufs -- especially for serialization, and to a lesser extent deserialization -- even when you're parsing the entire message (they shine comparatively even more if you're extracting sub-components of a message). One of them was made by the original inventor/lead-dev of the protobuf team, and he learned from some of his mistakes. "Proper" binary formats (like those, though they're by no means the only options) take into account data dependencies and other features of real hardware and are much closer to being limited by RAM bandwidth instead of CPU cycles.
That's sort of where I've landed too. Protobufs would seem to fit the problem area well, but in practice the space between "big-system non-performance-sensitive data transfer metaformat"[1] and "super-performance-sensitive custom binary parser"[2] is... actually really small.
There are just very few spots that actually "need" protobuf at a level of urgency that would justify walking away from self-describing text formats (which is a big, big disadvantage for binary formats!).
[1] Something very well served by JSON
[2] Network routing, stateful packet inspection, on-the-fly transcoding. Stuff that you'd never think to use a "standard format" for.
Perhaps surprisingly, I think microcontrollers may be a place where Protobufs are not a bad fit. Using something like Nanopb [1] gives you the size/speed/flexibility advantages of protocol buffers without being too heavyweight. It’ll be a bit slower than your custom binary protocol, but it comes with quite a few advantages, depending on the context.
We carry a nanopb integration in Zephyr. And even there... meh. It's true that there are some really bad binary protocols in the embedded world. And protobufs are for sure a step up from a median command parser or whatever. And they have real size advantages vs. JSON for tiny/sub-megabyte devices, which is real.
But even there, I find that really these are very big machines in a historical sense. And text parsing is really not that hard, or that big. The first HTTP server was on a 25MHz 68040!
Just use JSON. Anything else in the modern world needs to be presumed to be premature optimization absent a solid analysis with numbers.
If you use JSON in an embedded capacity and you're stingy with your bytes, you can just send arrays as your main document.
There was this one MMO I played, where every packet was just a space separated string, with a fancy variable length number encoding that let them store two digits in a single character.
Apart from being text format, I'm not sure how well JSON-RPC handles doubles vs long integers and other types, where protobuf can be directed to handle them appropriately. That is a problem in JSON itself, so you may neeed to encode some numbers using... "string"
I'd say the success of REST kind of proves that's something that for the most part can be worked around. Often comes down to the JSON codec itself, many codecs will allow unmarshalling/marshalling fields straight into long int types.
Also JS now has BigInt types and the JSON decoder can be told to use them. So I'd argue it's kind of a moot point at this stage.
Sure, but you can work around gRPC's issues too—"workable" might be the only bar that matters in practice, but it's a remarkably low bar.
The risk with JSON is that too many systems understand it, and intermediate steps can mess up things like numeric precision as well as being inconsistent about handling things out of spec (field order, duplicate fields... etc). This definitely bites people in practice—I saw an experience report on that recently, but can't find the link just now :/
JS having BigInt type has nothing to do with JSON. Backend languages have had BigInt types forever. It isn't relevant to JSON as a format.
Just one example: Azure Cosmos DB stores JSON documents. If you try to store integers larger than 53 bits there those integers will be silently rounded to 53 bits. I know someone who got burned by this very badly loosing their data; I was able to explain them exactly why...
JSON basically does not define what a number is; and that is a disaster for it as an API format.
It's not, because some middleman (library, framework, etc.) would assume that JSON is really about sending integers as doubles, hence you are getting only 53 or was it 54 bits precision, and then you end up sending an integer as "string" - but then what is this really?
I get it, it's probably not a concern for a lot of applications, but when comes to science, games, data it's of big concern... and this excluding the fact that you have to convert back and forth that number a... number of times, and send it on the wire inefficiently - and also miss a way to send it more efficiently using gorilla encoding or something else like that.
JSON is great for a lot of things, but not for high throughput RPC.
While true, it's still a text and usually http/tcp based format; data -> json representation -> compression? -> http -> tcp -> decompression -> parsing -> data. Translating to / from a text just feels inefficient.
Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc.
Varlink (https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.