To be honest I kind of find myself drifting away from gRPC/protobuf in my recent projects. I love the idea of an IDL for describing APIs and a great compiler/codegen (protoc) but there's just soo many idiosyncrasies baked into gRPC at this point that it often doesn't feel worth it IMO.
Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc.
Varlink (https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.
My favorite serde format is Msgpack since it can be dropped in for an almost one-to-one replacement of JSON. There's also CBOR which is based on MsgPack but has diverged a bit and added and a data definition language too (CDDL).
Take JSON-RPC and replace JSON with MsgPack for better handling of integer and float types. MsgPack/CBOR are easy to parse in place directly into stack objects too. It's super fast even on embedded. I've been shipping it for years in embedded projects using a Nim implementation for ESP32s (1) and later made a non-allocating version (2). It's also generally easy to convert MsgPack/CBOR to JSON for debugging, etc.
There's also an IoT focused RPC based on CBOR that's an IETF standard and a time series format (3). The RPC is used a fair bit in some projects.
What I really like about protobuf is the DDL. Really clear schema evolution rules. Ironclad types. Protobuf moves its complexity into things like default zero values, which are irritating but readily apparent. With json, it's superficially fine, but later on you discover that you need to be worrying about implementation-specific stuff like big ints getting mangled, or special parsing logic you need to set default values for string enums so that adding new values doesn't break backwards compatibility. Json-schema exists but really isn't built for these sorts of constraints, and if you try to use json-schema like protobuf, it can get pretty hairy.
Honestly, if protobuf just serialized to a strictly-specified subset of json, I'd be happy with that. I'm not in it for the fast ser/de, and something human-readable could be good. But when multiple services maintained by different teams are passing messages around, a robust schema language is a MASSIVE help. I haven't used Avro, but I assume it's similarly useful.
Software lives forever. You have to take the long view, not the "rn" view. In the long view, NFS's XDR or ASN.1 are just fine and could have been enough, if we didn't keep reinventing things.
It's mind-blowing to think XDR / ONC RPC V2 were products of the 1980s, and that sitting here nearly forty years later we are discussing the same problem space.
Probably the biggest challenge with something like XDR is it's very hard to maintain tooling around it long-term. Nobody wants to pay for forty years of continuous incremental improvement, maintenance, and modernization.
Long term this churn will hopefully slow down, it's inevitable as we collectively develop a solid set of "engineering principles" for the industry.
I'm actually super excited to see how https://www.sovereign.tech turns out long-term. Germany has a lot of issues with missing the boat on tech, but the sovereign tech fund is such a fantastic idea.
Buf seems really nice, but I'm not completely sure what's free and what's not with the Buf platform, so I'm hesitant to make it a dependency for my little open source side project ideas. I should read the docs a bit more.
Buf CLI itself is licensed under a permissive Apache 2.0 License [0]. Since Buf is a compiler, its output cannot be copyrighted (similar to proprietary or GPL licensed compilers). DISCLAIMER: I am not a lawyer.
Buf distinguishes a few types of plugins: the most important being local and remote.
Local plugins are executables installed on your own machine, and Buf places no restrictions on use of those.
Remote plugins are hosted on BSR (Buf Schema Registry) servers [1], which are rate limited. All remote plugins are also available as local plugins if you install them.
It's worth to mention that the only time I've personally hit the rate limits of remote plugins is when I misconfigured makefile dependencies to run buf on every change of my code, instead of every change of proto definitions. So, for most development purposes, even remote plugins should be fine.
Additionally, BSR also offers hosting of user proto schemas and plugins, and this is where pricing comes in [2].
> I love the idea of an IDL for describing APIs and a great compiler/codegen (protoc)
Me too. My context is that I end up using RPC-ish patterns when doing slightly out-of-the-ordinary web stuff, like websockets, iframe communications, and web workers.
In each of those situations you start with a bidirectional communication channel, but you have to build your own request-response layer if you need that. JSON-RPC is a good place to start, because the spec is basically just "agree to use `id` to match up requests and responses" and very little else of note.
I've been looking around for a "minimum viable IDL" to add to that, and I think my conclusion so far is "just write out a TypeScript file". This works when all my software is web/TypeScript anyway.
Now that's an interesting thought, I wonder if you could use a modified subset of TypeScript to create a IDL/DDL for JSON-RPC. Then compile that schema into implementations for various target languages.
Same; at my previous job for the serialisation format for our embedded devices over 2G/4G/LoRaWAN/satellite I ended up landing on MessagePack, but that was partially because the "schema"/typed deserialisation was all in the same language for both the firmware and the server (Nim, in this case) and directly shared source-to-source. That won't work for a lot of cases of course, but it was quite nice for ours!
State of the art for both gzipped json and protobufs is a few GB/s. Details matter (big strings, arrays, and binary data will push protos to 2x-10x faster in typical cases), but it's not the kind of landslide victory you'd get from a proper binary protocol. There isn't much need to feel like you're missing out.
The big problem with Gzipped JSON is that once unzipped, it's gigantic. And you have to parse everything, even if you just need a few values. Just the memory bottleneck of having to munch through a string in JSON is going to slow down your parser by a ton. In contrast, a string in Protobuf is length-encoded.
5-10x is not uncommon, and that's kissing an order of magnitude difference.
> have to parse everything, even for just a few values
That's true of protobufs as much as it is for json, except for skipping over large submessages.
> memory bottleneck
Interestingly, JSON, gzipped JSON, and protobufs are all core-bound parsing operations. The culprit is, mostly, a huge data dependency baked into the spec. You can unlock another multiplicative 10x-30x just with a better binary protocol.
> 5-10x is not uncommon
I think that's in line with what I said. You typically see 2x-10x, sometimes more (arrays of floats, when serialized using the faster of many equivalent protobuf wire encodings, are pathologically better for protos than gzipped JSON), sometimes less. They were aware of and worried about some sort of massive perf impact and choosing to avoid protos anyway for developer ergonomics, so I chimed in with some typical perf numbers. It's better (perf-wise) than writing a backend in Python, but you'll probably still be able to measure the impact in real dollars if you have 100k+ QPS.
Yeah this is something people don't seem to want to get into their heads. If all you care is minimizing transferred bytes, then gzip+JSON is actually surprisingly competitive, to the point where you probably shouldn't even bother with anything else.
Meanwhile if you care about parsing speed, there is MessagePack and CBOR.
If any form of parsing is too expensive for you, you're better off with FlatBuffers and capnproto.
Finally there is the holy grail: Use JIT compilation to generate "serialization" and "deserialization" code at runtime through schema negotiation, whenever you create a long lived connection. Since your protocol is unique for every (origin, destination) architecture+schema tuple, you can in theory write out the data in a way that the target machine can directly interpret as memory after sanity checking the pointers. This could beat JSON, MessagePack, CBOR, FlatBuffers and capnproto in a single "protocol".
And then there is protobuf/grpc, which seems to be in this weird place, where it is not particularly good at anything.
Protobufs have a massive data dependency baked into the wire format, turning parsing into an intensive core-bound problem.
Interestingly, they're not usually smaller than gzipped JSON either (the compression it has built-in is pretty rudimentary), so if you don't compress it and don't have a stellar network you might actually pay more for the total transfer+decode than gzipped JSON, despite usually being somewhat faster to parse.
The docs [0] are fairly straightforward. I'll spit out a little extra data and a few other links in case it's helpful. If this is too much or not enough text, feel free to ask followup questions.
As far as data dependencies are concerned, you simply can't parse a byte till you've parsed all the preceding bytes at the same level in a message.
A naive implementation would (a) varint decode at an offset, (b) extract the tag type and field index, (c) use that to parse the remaining data for that field, (c1) the exact point in time you recurse for submessages doesn't matter much, but you'll have to eventually, (d) skip forward the length of the field you parsed, (e) if not done then go back to (a).
You can do better, but not much better, because the varints in question are 8 bytes, requiring up to 10 bytes on the wire, meaning AVX2 SIMD shenanigans can only guarantee that you parse 3 varints at a time. That's fine and dandy, except most fields look like 2 varints followed by some binary data, so all you're really saying is that you can only parse one field at a time and still have to skip forward an unpredictable amount after a very short number of bytes/instructions.
If you have more specialized data (e.g., you predict that all field indexes are under 32 and all fields are of type "LENGTH"), then there are some tricks you can do to speed it up a bit further. Doing so adds branches to code which is already very branchy and data-dependent though, so it's pretty easy to accidentally slow down parsing in the process.
Something close to the SOTA for varint decoding (a sub-component of protobuf parsing) is here [1]. It's quite fast (5-10 GB/s), but it relies on several properties that don't actually apply to the protobuf wire format, including that their varints are far too small and they're all consecutively concatenated. The SOTA for protobuf parsing is much slower (except for the sub-portions that are straight memcopies -- giant slices of raw data are fairly efficient in protos and not in JSON).
This isn't the best resource [2], but it's one of many similar examples showing people not finding protos substantially faster in the wild, partly because their protos were bigger than their json objects (and they weren't even gzipping -- the difference there likely comes from the tag+length prefix structure being more expensive than delimiters, combined fixed-width types favoring json when the inputs are small). AFAICT, their json library isn't even simdjson (or similar), which ought to skew against protos even further if you're comparing optimal implementations.
In terms of protos being larger than gzipped json, that's just an expected result for almost all real-world data. Protobuf adds overhead to every field, byte-compresses some integers, doesn't compress anything else, and doesn't bit-compress anything. Even if your devs know not to use varint fields for data you expect to be negative any fraction of the time, know to use packed arrays, ..., the ceiling on the format (from a compression standpoint) is very low unless your data is mostly large binary blobs that you can compress before storing in the protobuf itself.
For a few other random interblags comparisons, see [3], [4]. The first finds protos 3x-6x faster (better for deserializing than serializing) compared to json. The second finds that protos compress better than json, but also that compressed json is much smaller than ordinary protos for documents more than a few hundred bytes (so to achieve the size improvements you do have to "cripple" protos by compressing them).
If you start looking at the comparisons people have done between the two, you'll find results largely consistent with what I've been saying: (1) Protos are 2x-10x faster for normal data, (2) protos are usually larger than gzipped json, (3) protos are sometimes slower than gzipped JSON, (4) when you factor in sub-par networks, the total transfer+decode time can be much worse for protos because of them being larger.
As a fun experiment, try optimizing two different programs. Both operate on 1MB of pseudo-random bytes no greater than 10. Pick any cheap operation (to prevent the compiler from optimizing the iteration away) like a rolling product mod 256, and apply that to the data. For the first program (simulating a simplified version of the protobuf wire format), treat the first byte as a length and the next "length" bytes as data, iterating till you're done. For the second, treat all bytes as data. Using a system's language on any modern CPU, you'll be hard-pressed to get an optimized version of the length-prefixed code even as fast as 10x slower than an un-optimized version of the raw data experiment.
Cap'n proto and flatbuffers (whether gzipped or not), as examples, are usually much faster than both JSON and protobufs -- especially for serialization, and to a lesser extent deserialization -- even when you're parsing the entire message (they shine comparatively even more if you're extracting sub-components of a message). One of them was made by the original inventor/lead-dev of the protobuf team, and he learned from some of his mistakes. "Proper" binary formats (like those, though they're by no means the only options) take into account data dependencies and other features of real hardware and are much closer to being limited by RAM bandwidth instead of CPU cycles.
That's sort of where I've landed too. Protobufs would seem to fit the problem area well, but in practice the space between "big-system non-performance-sensitive data transfer metaformat"[1] and "super-performance-sensitive custom binary parser"[2] is... actually really small.
There are just very few spots that actually "need" protobuf at a level of urgency that would justify walking away from self-describing text formats (which is a big, big disadvantage for binary formats!).
[1] Something very well served by JSON
[2] Network routing, stateful packet inspection, on-the-fly transcoding. Stuff that you'd never think to use a "standard format" for.
Perhaps surprisingly, I think microcontrollers may be a place where Protobufs are not a bad fit. Using something like Nanopb [1] gives you the size/speed/flexibility advantages of protocol buffers without being too heavyweight. It’ll be a bit slower than your custom binary protocol, but it comes with quite a few advantages, depending on the context.
We carry a nanopb integration in Zephyr. And even there... meh. It's true that there are some really bad binary protocols in the embedded world. And protobufs are for sure a step up from a median command parser or whatever. And they have real size advantages vs. JSON for tiny/sub-megabyte devices, which is real.
But even there, I find that really these are very big machines in a historical sense. And text parsing is really not that hard, or that big. The first HTTP server was on a 25MHz 68040!
Just use JSON. Anything else in the modern world needs to be presumed to be premature optimization absent a solid analysis with numbers.
If you use JSON in an embedded capacity and you're stingy with your bytes, you can just send arrays as your main document.
There was this one MMO I played, where every packet was just a space separated string, with a fancy variable length number encoding that let them store two digits in a single character.
Apart from being text format, I'm not sure how well JSON-RPC handles doubles vs long integers and other types, where protobuf can be directed to handle them appropriately. That is a problem in JSON itself, so you may neeed to encode some numbers using... "string"
I'd say the success of REST kind of proves that's something that for the most part can be worked around. Often comes down to the JSON codec itself, many codecs will allow unmarshalling/marshalling fields straight into long int types.
Also JS now has BigInt types and the JSON decoder can be told to use them. So I'd argue it's kind of a moot point at this stage.
Sure, but you can work around gRPC's issues too—"workable" might be the only bar that matters in practice, but it's a remarkably low bar.
The risk with JSON is that too many systems understand it, and intermediate steps can mess up things like numeric precision as well as being inconsistent about handling things out of spec (field order, duplicate fields... etc). This definitely bites people in practice—I saw an experience report on that recently, but can't find the link just now :/
JS having BigInt type has nothing to do with JSON. Backend languages have had BigInt types forever. It isn't relevant to JSON as a format.
Just one example: Azure Cosmos DB stores JSON documents. If you try to store integers larger than 53 bits there those integers will be silently rounded to 53 bits. I know someone who got burned by this very badly loosing their data; I was able to explain them exactly why...
JSON basically does not define what a number is; and that is a disaster for it as an API format.
It's not, because some middleman (library, framework, etc.) would assume that JSON is really about sending integers as doubles, hence you are getting only 53 or was it 54 bits precision, and then you end up sending an integer as "string" - but then what is this really?
I get it, it's probably not a concern for a lot of applications, but when comes to science, games, data it's of big concern... and this excluding the fact that you have to convert back and forth that number a... number of times, and send it on the wire inefficiently - and also miss a way to send it more efficiently using gorilla encoding or something else like that.
JSON is great for a lot of things, but not for high throughput RPC.
While true, it's still a text and usually http/tcp based format; data -> json representation -> compression? -> http -> tcp -> decompression -> parsing -> data. Translating to / from a text just feels inefficient.
The opaque API brings some niceties that other languages have, specifically about initialization. The Java impl for protobuf will never generate a NullPointerException, as calling `get` on a field would just return the default instance of that field.
The Go OpenAPI did not do this. For many primative types, it was fine. But for protobuf maps, you had to check if the map had been initialized yet in Go code before accessing it. Meaning, with the Opaque API, you can start just adding items to a proto map in Go code without thinking about initialization. (as the Opaque impl will init the map for you).
This is honestly something I wish Go itself would do. Allowing for nil maps in Go is such a footgun.
The Java impl for protobuf will never generate a NullPointerException, as calling `get` on a field would just return the default instance of that field.
This was a mistake. You still want to check whether it was initialized most of the time, and when you do the wrong thing it's even more difficult to see the error.
Depends on your use. If you are parsing a message you just received, I agree that you want to do a "has" check before accessing a field. But when constructing a message, having to manually create all the options is really annoying. (I do love the java builder pattern for protos).
But I do know the footgun of calling "get" on a Java Proto Builder without setting it, as that actually initializes the field to empty, and could call it to be emitted as such.
Such are the tradeoffs. I'd prefer null-safety to accidental field setting (or thinking a field was set, when it really wasn't).
I too enjoy defining my default attribute values at the protocol level, so they can never, ever be changed, and then defensively coding library functions in case the arguments are not constructed under the same semantic assumptions as I had years ago.
Also everything needs to be marked optional or you'll need to restart the whole service mesh to change the schema.
Default values in proto are considered an anti pattern and was removed from the language in proto3.
Agreed that protocol level required is bad, and it's with proto3 made optional the default.
Our team enforces required fields via proto annotations, which can be much more flexible (we have transitional states to be able to upgrade or downgrade the check)
It's so fun to watch go devs rediscover all the patterns that they so happily threw out in the beginning. It's like watching a person grow up from a sunny little kid to a mature disgruntled alcoholic.
An alternative explanation is that non-go people got their hands on go and complain that go is not x or y.
Like with generics. Now they’re in go. They’re not great, they have some sense but they may as well not exist as far as I’m concerned. I find them pretty useless anyway without lower and upper type bounds.
„go can’t be taken seriously because it doesn’t have generics” has been a mantra for years before they finally arrived. there must have been a ton of them.
That does not mean that generics were somehow forced in against the wishes of the Go Team. Their position has always been that they just need to find a good design.
And everybody else laughed at the notion that there's a lack of good design for generics, given that all other mainstream languages have had them for literally decades, with many different takes to choose from.
Given that Go eventually ended up with a very mundane take on generics, in retrospect, it is not at all clear what precluded this same design from being adopted from the get go rather than dragging their feet on it.
> And everybody else laughed at the notion that there's a lack of good design for generics, given that all other mainstream languages have had them for literally decades, with many different takes to choose from.
Yes, but they weren't good; half of the Java spec and implementations has to deal with its implementation of generics, and the author of Java's generics (Martin Odersky) then moved on to create the even more complicated Scala language. I have this comment [0] and its linked "Java generics FAQ" [1] that is exemplary of its complexity bookmarked for this very occasion.
They ended up with a mundane take on generics because they dragged their feet on it. It's like that quote, "I would have written a shorter letter, but did not have the time." If generics were added without waiting for enough demand, use cases and consideration, it would have ended up weighing them (spec, language, compiler, runtime, etc) down for decades to come. Remember that Go is in it for the long haul.
Java is one example out of many, and even in that case the limitations of their generics are largely caused by the desire for backwards compatibility, and it's a well-understood tradeoff.
Go ended up with a mundane take on generics because that is consistent with the design philosophy of the language as a whole. My point is that their mundane take on generics was already well-known and well-understood long before Go itself was even a thing, never mind all the rhetoric about "we don't have generics because we don't know how to do them well". If they ended up with some kind of novel arrangement that avoided complexity issues etc present in other generic type system, I would buy that argument. But when they ended up with the design that was there all along, it doesn't really add up.
> The Java impl for protobuf will never generate a NullPointerException, as calling `get` on a field would just return the default instance of that field.
It's interesting, to everyone but but the mega shops like Google, protobuf is a schema declaration tool. To the megashops its a performance tool.
For most of my projects, I use a web-framework I built on protobuf over the years but slowly got rid of a lot of the protobufy bits (besides the type + method declarations) and just switched to JSON as the wire format. http2, trailing headers, gigantic multi-MB files of getters, setters and embedded binary representations of the schemas, weird import behaviors, no wire error types, etc were too annoying.
Almost every project I've tracked that tries to solve the declarative schema problem seems to slowly die. Its a tough problem an opinionated one (what to do with enums? sum types? defaults? etc). Anyone know of any good ones that are chugging along? OpenAPI is too resty and JSONSchema doesn't seem to care about RPC.
> It's interesting, to everyone but but the mega shops like Google, protobuf is a schema declaration tool
There are lots of other benefits for non performance-oriented teams and projects: the codegen makes it language independent and it's pretty handy that you can share a single data model across all layers of your system.
If you don't care about the wire format, the standard JSON representation makes it pair well with JSON native databases, so you can get strict schema management without the need need for any clunky ORM.
That's assuming JSON native databases are fine for your use case though, but in practice it's only good for storing documents that don't need to be edited/queried much by the backend storing them.
> in practice it's only good for storing documents that don't need to be edited/queried much by the backend storing them
Why aren't they good for that? They can have very high write throughput, they don't require ORMs, and they can be indexed and queried using standard database methods like the SQL language.
You can even enforce strict schemas on them if you want to, just as you would with an RDBMS.
At scale, the performance gains can be dramatic. For example, moving a json web service to CBOR, I was able to squeeze 15% more throughput out of existing hardware. When you’re dealing with hundreds, if not millions of requests per minute this can be financially prudent.
In my case I was using CBOR, not protobuf. The effect could be similar though when moving from json to protobuf. You get a much more efficient serialization and deserialization. This reduces CPU demand on server side for sending response payload and for client to receive the payload.
Coral is a schema definition language, yes. But it’s also a full rpc ecosystem.
Smithy at this point is only really an IDL that (in most cases, at least before I left) is “only” used to generate Coral models and then transitively Coral clients and services. The _vast_ majority of Amazon is still on “native” Coral
There's a bit of boilerplate in there if you want to use it for a naive implementation, but I don't find it exceedingly resty.
From my POV, using protobuf as a schema declaration tool (as opposed to being a performance tool) is blind follower behaviour. Getting over all the hurdles doesn't seem worth it for the payoff, and it only becomes less valuable when compared to all the OpenAPI tooling you could be enabling instead.
This being for a web-based problem, where we're solving schema declaration.
Protobuf 3 was bending over backwards to try to make the Go API make sense, but in the process it screwed up the API for C++, with many compromises. Then they changed course and made presence explicit again in proto 3.1. Now they are saying Go gets a C++-like API.
What I'd like is to rewind the time machine and undo all the path-dependent brain damage.
I wasn't there for the debate, but was there for the reversal. I don't remember there being anything explicitly said about it. The only thing I can think of is that I know of some important projects that couldn't migrate to proto3 because of this implicit field issue. So some people were still writing new code with proto2.
And... most of the important projects are still using proto2 and there is no realistic way to do interop between those two formats. IIRC, you cannot use proto2 enum in proto3 for some obscure technical reason! This creates a backsliding issue; you cannot migrate old protos with thousands of different messages, fields and enums where new proto2 messages to use them are added everyday.
This is an important distinction from the proto1->proto2 migration, which was in a much better compatibility situation yet still took years to complete. AFAIK, this is the main reason why the proto team decided to create a superset approach (edition) so migration can be handled as a flag at each message level.
I work mainly in Python, it's always seemed really bad that there are 3 main implementations of Protobufs, instead of the C++ being the real implementation and other platforms just dlopen'ing and using it (there are a million software engineering arguments around this; I've heard them all before, have my own opinions, and have listened to the opinions of people I disagree with). It seems like the velocity of a project is the reciprocal of the number of independent implementations of a spec because any one of the implementations can slow down all the implementations (like what happened with proto3 around required and optional).
From what I can tell, a major source of the problem was that protobuf field semantics were absolutely critical to the scaling of google in the early days (as an inter-server protocol for rapidly evolving things like the search stack), but it's also being used as a data modelling toolkit (as a way of representing data with a high level of fidelity). And those two groups- along with the multiple language developers who don't want to deal with native code- do not see eye to eye, and want to drive the spec in their preferred direction.
(FWIW nowadays I use pydantic for type descriptions and JSON for transport, but I really prefer having an external IDL unrelated to any specific programming language)
> It seems like the velocity of a project is the reciprocal of the number of independent implementations of a spec because any one of the implementations can slow down all the implementations (like what happened with proto3 around required and optional).
Velocity and stability/maturity are in tension, sure. I think for a foundational protocol like protobuf you want the stability and reliability that come from multiple independent implementations more than you want it to be moving fast and breaking things.
Additionally calling C++ from other languages is a pain and you're forced to make a bridge C API. Doing so from Go is even less than ideal from what I gather. It requires using cgo and forces Go to interface with C call stacks, slows down the compilation, etc.
You'll be thrilled to hear about upb then, which was designed to be embeddable to power other languages without a from-scratch implementation - and now powers python protos.
I still use proto2 if possible. The syntactic sugar around `oneof` wasn't nice enough to merit dealing with proto3's implicit presence -- maybe it is just because I learned proto2 with C++ and don't use Go, but proto3 just seemed like a big step back and introduced footguns that weren't there before. Happy to hear they are reverting some of those finally.
Nice to see my comments on their proto3 design doc vindicated, lol. There were a lot of comments on that doc, far more than what you'd usually see. Some of those comments dealt with the misguided decision to basically drop nullability (that is, the `has_` methods) that proto2 had. The team then just deleted all the comments and disabled commenting on the doc and proceeded with their original design much to the consternation of their primary stakeholders.
> syntax = "proto2" uses explicit presence by default
> syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword
> edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default
The root of the problem seems to be go's zero-values. It's like putting makeup on a pig, your get rid of null-panics, but the null-ish values are still everywhere, you just have bad data creeping into every last corner of your code. There is no amount of validation that can fix the lack of decoding errors. And it's not runtime errors instead of compile-time errors, which can be kept in check with unit tests to some degree. It's just bad data and defaulting to carry on no matter what, like PHP back in the day.
> It's like putting makeup on a pig, your get rid of null-panics
How so? In Go, nil is the zero value for a pointer and is ripe for panic just like null. Zero values do not avoid that problem at all, nor do they intend to.
Ill give you that nil is a fine default for pointers, and pointer like things (interfaces, maps, slices). Its mostly fine to use empty string. However 0 has semantic meaning for just about every serialized numeric type I've ever encountered. The zero value also does really poorly for PUT style apis, "did the user forget to send this or did they mean to set this field to empty string" is very poorly expressed in Go and often has footguns around adding new fields.
In Go, you would use additional bits to identify whether the zero value is valid/present. You would not use the zero value of an integer, or other type, alone, unless the zero value truly has special meaning and can stand on its own; an uncommon case for integers, as you point out.
Unfortunately, there is no escaping the extra bits. Not a terribly big deal for a large, powerful machine with lots of memory, but might be a big deal over constrained networks. Presumably that is why proto3 tried to save on the amount of data being transferred. It adds up at Google scale. But it did eventually walk back on that idea, making the extra data opt-in.
I'm not convinced at all that data-size is the reason for any of this. You would also be sending more data on any type that is not a primitive, i.e. any type that is bigger then the sentinel null value, you would have to send the full blown struct.
As is often the case, Go's designed anachronism creates more problem than it solves: had Go had a modern, expressive type system, rather than staying with one from the 70s, this problem would never exists.
The slice type is an implicit struct, in the shape:
struct {
data uintptr
len int
cap int
}
Which is usable when the underlying memory is set to zero. So its zero value is really an empty slice. Most languages seem to have settled on empty slice, array, etc. as the initialized state just the same. I find it interesting you consider that the worst case.
Maps are similar, but have internal data structures that require initialization, thus cannot be reliably used when zeroed. This is probably not so much a performance optimization as convention. You see similar instances in the standard library. For example:
var file os.File
file.Read([]byte{}) // panics; file must be initialized first.
> Maps are similar, but have internal data structures that require initialization, thus cannot be reliably used when zeroed.
Its a performance thing, the map structure requires a heap allocation even with zero elements. This is because the map structure is type generic without being fully monomorphized.
The heap allocation could transparently happen at use. The only additional performance overhead that would bring is the conditional around if it has already been initialized or not, which is negligible for modern computers. Go is not obsessed with eking every last bit of performance anyway.
But, that is not the convention. It is well understood that if a struct needs allocations that you must first call a constructor function (e.g. package.NewFoo). It is reasonable to expect that map would stick to the same convention.
It should be noted that slices and maps are completely opposite ends of how they behave in relation to nil in Go. A nil slice is just an empty slice, there is no operation you could do with one that will fail if done with the other. In contrast, a nil map doesn't support any operation whatsoever, it will panic on doing anything with it.
You're quite mistaken. You can use len() on both nil maps and slices, and it will return zero (as with empty ones). Panic occurs on both nil assignments. But then, access only panics on nil slices - nil maps produce the zero value. It's horrible.
Right, forgot that len() works on nil maps, and I was really not aware that reading from a nil map is not an error, that's crazy.
For the nil slice though what I said remains true: a nil slice is the same thing as an empty slice. Of course reading from or writing to its first element panics, given that it doesn't have any elements. The same would have happened if you had initialized it as `var ns []int = make([]int, 0)` or `ns := []int{}`.
Ah now I get it, yes you're right, they behave the same.
Ironically, just today I read that in the next release they will add some JSON annotation to distinguish nil slices (omitzero): https://tip.golang.org/doc/go1.24
I don't think the reason for zero values has anything to do with "avoiding null panics". If you want to inline the types, that is avoid using most of your runtime on pointer chasing, you can't universally encode a null value. If I'm unclear, ask yourself: What would a null int look like?
If what you wanted was to avoid null-panics, you can define the elementary operations on null. Generally null has always been defined as aggressively erroring, but there's nothing stopping a language definition from defining propagation rules like for float NaN.
Sorry, I don't follow you. If you don't have zero values, you either have nulls and panics, or you have some kind of sum-type á la Option<T> and cannot possibly construct null or zero-ish values.
Is there a way to have your cake and eat it too, and are there real world examples of it?
You're thinking in abstract terms, I'm talking about the concrete implementation details. If we, just as an example, take C. and int can never be NULL. It can be 0, compilers will sometimes tell you it's "uninitialized", but it can never be NULL. all possible combinations of bit patterns are meaningfully int.
Pointers are different in that we've decided that the pattern where all bits are 0 is a value that indicates that it's not valid. Note that there's nothing in the definition of the underlying hardware that required this. 0 is an address just like any other, and we could have decided to just have all pointers mean the same thing, but we didn't.
The NULL is just a language construct, and as a language construct it could be defined in any way you want it. You could defined your language such that dereferencing NULL would always return 0. You could decide that doing pointer arithmetic with NULL would yield another NULL. At the point you realize that it's just language semantics and not fundamental computer science, you realize that the definition is arbitrary, and any other definition would do.
As for sum-types. You can't fundamentally encode any more information into an int. It's already completely saturated. What a sumtype does, at a fundamental level, is to bundle your int (which has a default value) with a boolean (which also has a default value) indicating if your int is valid. There's some optimizations you can do with a "sufficiently smart compiler" but like auto vectorization, that's never going to happen.
I guess my point can be boiled down to the dual of the old C++ adage. Resource Allocation is NOT initialization. RAINI.
No, there isn't. It is just other versions of the same problem with people pretending it is somehow different.
People generally like to complain about NULL/nil whatever, but they rarely think about what the alternatives mean and what arrangements are completely equivalent. No matter what you do, you have to put some thought into design. Languages can't do the design work for programmers.
There is a way to have your cake and eat it too: rust.
In rust, you have:
let s = S{foo: 42, ..Default::default()};
You just got all the remaining fields of 'S' set to "zero-ish" values, and there's no NPEs.
The way you do this is by having types opt in to it, since zero values only make sense in some contexts.
In go, the way to figure out if a type has a meaningful zero value is to read the docs. Every type has a zero value, but a lot of them just nil-pointer-exception or do something completely nonsensical if you try to use them.
In rust, at compiletime you can know if something implements default or not, and so you can know if there's a sensible zero value, and you can construct it.
Go doesn't give you your cake, it gives you doc comments saying "the zero value is safe to use" and "the zero value will cause unspecified behavior, please don't do it", which is clearly not _better_.
> There is a way to have your cake and eat it too: rust.
Suppose my cake is that I have a struct A which holds a value, that doesn't have a default value, from your library B. Suppose that at the time I want to allocate A I don't yet have the information I need to initialize B, but I also know that I won't need B before I do have that information and can initialize it. In simple terms. I want to allocate A, which requires allocating B, but I don't want to initialize B, yet.
What do I do?
If you answer involves Option<B> then you're asking me to to grow my struct for no gain. That is clearly not _better_.
Doesn't Rust have explicit support for uninitialized memory, using the borrow checker to make sure you don't access it before initializing it? Or does that just work for local variables, not members of structs?
More specifically, it could result in undefined behavior, if a panic happens between the allocation and initialization (i.e., it was allocated, not initialized, panicked, and something observed the incomplete struct after the panic). Alternatively, the allocation would always have to leak on panic, or the struct would have to be deallocated without a destructor running.
I agree that rust, with Option and Default, is the only right choice - at least from what I've tried. Elm for example has Option but nothing like Default, so sometimes it's tedious that you have to repeat a lot of handmade defaults, or you're forced to use constructor functions everywhere. But at least the program is correct!
Go is like PHP in regards to pushing errors forward. You simply cannot validate everything at every step. Decoding with invariants is the right alternative.
What is with Rust evangelicals shitting up Go posts? Shut up and go away! Go talk to other Rust users about it if you love it so much!
it's for different things!
the things I build in Go simply do not need to be robust in the way Rust requires everything to be, and it would be much more effort to use Rust in those problem domains
Is Go a more crude language? maybe! but it lets me GET SHIT DONE and in this case worse really is better.
All I know is that I've spent less time over the last ten years writing Go dealing with NPEs than I have listening to Rust users complaining about them!
if you love Rust so much, YOU use it then! We like Go, in threads about Go. I might like Rust too, in the same way I like my bicycle and my car, if only the cyclists would shut up about how superior their choices are
Exactly my point, you have to opt-in, and in practice you only do precisely where it's actually necessary. Which is completely different than "every single type can be a [null | zero value]". You cannot possibly construct some type A (that is not Option<A> or A@nullable or whatever) without populating it correctly.
Of course you need some way to represent "absence of a value", the matter is how: simple but incorrect, or complex but correct. And, simple/complex here can mean both the language (so performance tradeoff), and (initial) programmer ergonomics.
That's why I ask if you can have your cake and eat it too, the answer is no. Or you'll get sick sooner than later, in this case.
There is no way to store None | Some(x) in sizeof(x) bytes, for simple information theory reasons. What you can do is store between 1 and 8 optional fields with only 1 byte of overhead, by using a single bit field to indicate which of the optional fields is set or not (since no commonly used processor supports bit-level addressing, storing 1 extra bit still needs an entire extra byte, so the other 7 bits in that byte are "free").
> There is no way to store None | Some(x) in sizeof(x) bytes
That's subtlety incorrect. Almost all languages with NULLs in fact already do this, including C. On my machine sizeof(void*)=8, and pointers can in fact express Some(x)|None. The cost of that None is neither a bit not a byte, it's a single value. A singular bit pattern.
See the None that you talk about is grafted on. It wraps the original without interfacing with it. It extends the state by saying "whatever the value this thing has, its invalid". That's super wasteful. Instead of adding a single state, you've exploded the state space exponentially (in the literal sense).
I should have made that caveat: if X doesn't need all of the bits that it has, then yes, you can do this. But there is no way to know that this is the case for a generic type parameter, you can only do this if you know the semantics of the specific type you are dealing with, like the language does for pointer types.
I should also point out that in languages which support both, Option(int*) is a valid construct, and Some(nullptr) is thus not the same thing as None. There could even be valid reasons for needing to do this, such as distinguishing between the JSON objects {} and {"abc": null}. So you can't even use knowledge of built-in types to special-case your generic Option implementation. Even Option(Option(bool)) should be able to represent at least 4 distinct states, not 3: None, Some(None), Some(Some(true)), Some(Some(false)).
From what I remember, proto3 behavior happened to map to objective c since iOS maps coincidentally happened at around the same time so they could be loud.
It was partially reverted with proto3 optional and fully reverted finally. Go's implementation happened to come around the same time as proto3 so allowed struct access, despite behaving quite differently when accessing nil fields. That is also finally reverted. Hopefully more lessons already learned from the Java days will come sooner than later going forward...
Yes, as much as I love Go and love working with it every day. This inner workings of Go with zero-values has been an design issue that comes up again and again and again.
I hate this API and Go's handling of protocol buffers in general. Especially preparing test data for it makes for some of the most cumbersome and unwieldy files that you will ever come across. Combined with table driven testing you have thousands upon thousands of lines of data with an unbelievably long identifiers that can't be inferred (e.g. in array literals) that is usually copy pasted around and slightly changed. Updating and understanding all of that is a nightmare and if you miss a coma or a brace somewhere, the compiler isn't smart enough to point you to where so you get lines upon lines of syntax errors. But, being opaque has some advantages for sure.
I find that the best way to set up test cases, regardless of language, is usually to use string constants in the proto text format (https://protobuf.dev/reference/protobuf/textformat-spec/). For arrays, and especially for oneofs, it's way less verbose than how things are represented in Go, C++, or Java, and generally at least on par with the Python constructors. Maps are the only thing that suffer a bit, because they're represented as a list of key-val pairs (like in the wire format) instead of an actual map. Your language's compiler won't help you debug, but the parse-text-proto function can point to the source of the issue on a line/character level.
With Go generics - and equivalent in most other languages - you can write a 5-line helper function that takes a string in that format and either returns a valid proto value (using the generic type param to decide which type to unmarshal) or `t.Fatal()`s. You would never do this in production code, but as a way to represent hand-written proto values it's pretty hard to beat.
If your problem is humans making arbitrary and nonsensical decisions about how you can do your job, then you have a non-technical problem, and it's unlikely that any technical solution will solve it.
The generated Go code situation has always been wild to me. For example, every message embeds protoimpl.MessageState and a bunch of other types, which contain mutexes. That means proto structs cannot be copied or compared byte-for-byte like normal Go structs can.
For several years I used the GoGo Protobuf SDK. It was vastly superior to the awful Javaesque Go code that the official compiler generated. It allowed structs to be pure data structs, was much more performant, and supported a bunch of options to generate native-feeling, ergonomic Go code.
But the Google team refused to partake in any such improvements, and GoGo was shut down as the burden of following the upstream implementation became too big. [1]
I'm not an expert, but as far as I understand, the extra struct junk is mostly to avoid having a parallel set of types for metadata (including reflection). It's unclear to me why these can't simply be generated as internal types with some nice API on top.
Clearly the new field metadata adds to this extra information, and the Go team is moving in the opposite direction of what I thought the future was — they're doubling down on stuffing metadata into the structs, and making the structs bigger and even less wieldy.
I understand how this might make things more performant, but I was hoping this sort of thing could be solved with the type system, especially now that we have generics. For example, surely lazy field access could be done like this:
type Info struct {
User LazyProto[User]
}
userName := user.Get().Name
We put test inputs and outputs in testdata/test_name.in.textpb and testdata/test_name.out.textpb, respectively. Way nicer than defining both your inputs and your desired outputs in go code, even compared to not using protobuf at all, to the point where we occasionally write some proto definitions just for test inputs and outputs.
The testing practice I've seen is to have a testdata/ directory with a bunch of textprotos for different test cases. If you're using Bazel, just include the entire directory glob as data dependency for the unit tests. The test tables are essentially just appropriately named textproto filenames that are unmarshaled into the proto message to be tested.
Then again I've also seen people do these thousand line in-code literal string protos which really grind my gears.
I haven't used protocol buffers, but in general any kind of code generation produces awful code. I much prefer generating the machine spec (protocol buffers, in this case) from Go code rather than the other way around. It's not a perfect solution, but it's much better than dealing with generated code in my experience.
Generated code tends to look very formulaic, but it doesn’t have to be unreadable. As a primative it is incredibly powerful, and can be easier to maintain than alternatives. You definitely need good build tooling though. Ideally you won’t need to look at the generated code and can infer the interface from the input file.
It's pretty hard to generate formulaic Go code while ensuring that, for example, methods don't conflict with exported member names, or in cases when you're making identifiers from multiple words, that the identifiers for `FooBar Baz` does not conflict with the identifier for `Foo BarBaz` since both of these would naturally be rendered in Go as `FooBarBaz`. Or even how you model an optional type: in Go, an idiomatic optional may use nil for reference types, -1 for nonnegative integers, an empty string for nonempty strings, or a `(T, bool)` tuple. You can definitely make a code generator that does a decent job at modeling all of these things, but I've never seen one--they usually try to pick a rule that works for all cases, like `Foo_BarBaz` or using an extra indirection for optionals, which means all cases are not-idiomatic.
I recently used code-gen'd protobuf deser objects as the value type for an in-memory db and was considering flattening them into a more memory-efficient representation and using bitfields. That was for java though, not sure if they are doing the same thing there
Glad to see this change, for that use case it would've been perfect
Surprisingly I saw this on the front page mere minutes after deciding to use protobufs in my new project.
Currently I'm not quite sold on RPC since the performance benefits seem to show up on a much larger scale than what I am aiming for, so I'm using a proto schema to define my types and using protoc codegen to generate only JSON marshaling/unmarshaling + types for my golang backed and typescript frontend, with JSON transferred between the two using REST endpoints.
Seems to give me good typesafety along with 0 headache in serializing/deserializing after transport.
One thing I also wanted to do was generate SQL schemas from my proto definitions or SQL migrations but haven't found a tool to do so yet, might end up making one.
Would love to know if any HN folk have ideas/critique regarding this approach.
Oh, this is great. I just did an implementation in gRPC in Go whereby I had to churn through 10MB/s of data. I could not implement any kind of memory pool and thus I had a lot of memory allocation issues which lead to bad memory usage and garbage collection eating up my CPU.
Like the sibling said, there's a complimentary _builder struct generated with a Build() method. For instance, for the sample message in the blog post, here's the public API of the generated _builder:
Underscores are commonly used in names in generated code to avoid conflicts (this applies to all sorts of codegen, not just protobuf). You can easily have both Foo and FooBuilder messages in your protobuf. See also generated enum consts since day one of protobuf-gen-go.
Fair enough but now this name leaks out into code I have to type. I don't dig highly opinionated languages with tooling that complains at me over style guidelines breaking their own style guidelines to solve a problem that they themselves have created.
They painted themselves into a corner and their solution is for me to hit myself in the head with a hammer.
If a generated name clashes, then you just override the generated name.
And in this case, perhaps, the resistance and the ugliness of the solution should have made the Protobuf authors to pause for a bit and rethink it.
Several years ago, I used gogoprotobuf instead of regular Go protobuf. It uses normal by-value structures instead of pointers everywhere, with generated code for s11n instead of reflection. It worked about 10x faster that the regular protobuf.
why is code generation under-utilized? protobufs and other go tooling are great for code generation. Yet in practice i see few teams using it at scale.
Lots of teams creating rest / json APIs, but very few who use code generation to provide compile-time protection.
Code generation leaves a layer of abstraction between the API and the actual implementation which works great if that code generation is bug-free but if it's not, you're like... totally fucked. Most commonly people say you can read the generated code and step backwards but that's like saying you can read the compiled JavaScript and it's basically open source. That layer of abstraction is an underrated mental barrier.
Of course, code generation is still practical and I'm a lot more likely to trust a third-party writing a code generator like protobufs, OpenAPI specs, etc, but I would not trust an internal team to do so without a very good reason. I've worked on a few projects that lost hundreds of dev hours trying to maintain their code generator to avoid a tiny bit of copy/paste.
Code generation is under utilized because most people don't have a build system good enough for it. Traditional make is fine: you just define dependencies and rules. But a lot of people want to use language-specific build systems and these often don't have good support for code generation and dependency tracking for generated code.
Yet another subtlety is that when cross-compiling, you need to build the code generation tool for the local target always even though the main target could be a foreign architecture. And because the code generation tool and the main code could share dependencies, these dependencies need to be built twice for different targets. That again is something many build tools don't support.
The whole FlatBuffers toolchain is wildly immature compared to Protobuf. Last I checked, flatc doesn’t even have a plugin system - all code generators had to be upstreamed into the compiler itself.
Isn't protoc mostly the same? I mean I know the code generators are separate binaries (which is quite annoying frankly) but protoc needs to know of them all upstream to expose options like --go_out and --go_opt, right?
No, that's not the case - there's a standard to how protoc exposes plugin options as CLI options that doesn't require the compiler to know about each plugin. It's obtuse, annoying, and badly documented, but it's there :)
The whole situation is extra confusing because some of the core codegen plugins are built into protoc. That's a distribution and version-matching strategy from the Google team rather than a requirement of the plugin system. I'd very much like Google to ship the built-in plugins as standalone binaries, even if they're also bundled with protoc.
I'm not sure. We have a few private plugins at work and they work fine with regular protoc. I think the --go_out and friends are automatically created by the plugins. Something like, they declare themselves as "toto" and to protoc now has a --toto_out option.
If you work with both the ergonomic advantages of protobufs become quickly apparent - starting the first time you nest things a few times. Unless you are very very frequently not going to deserialize your entire messages and so can get huge benefits from the better selective-deser of only what a given consumer cares about at a certain time, I find using flatbuffers hard to justify.
Which binary data representation? If I'm sending a Java object, do you think a C program will be able to just use it? Or for that matter, do you think two different C++ implementations, maybe on different platforms, will use the same binary representation of a class object?
Great, now there's an API per struct/message to learn and communicate throughout the codebase, with all the getters and setters.
A given struct is probably faster for protobuf parsing in the new layout, but the complexity of the code probably increases, and I can see this complexity easily negating these gains.
> Great, now there's an API per struct/message to learn and communicate throughout the codebase, with all the getters and setters.
No, the general idea (and practical experience, at least for projects within Google) is that a codebase migrates completely from one API level to another. Only larger code bases will have to deal with different API levels. Even in such cases, your policy can remain “always use the Open API” unless you are interested in picking up the performance gains of the Opaque API.
message M {
string foo = 1;
}
message N {
M bar = 2;
}
I find (new(M)).Bar.Foo panicking pretty annoying. So I just made it a habit to m.GetBar().GetFoo() anyway. If m.GetBar().SetFoo() works with the new API, that would be an improvement.
There are some options like nilaway if you want static analysis to prevent you from writing this sort of code, but it's difficult to retrofit into an existing codebase that plays a little too fast and loose with nil values. Having code authors and code reviewers do the work is simpler, though probably less accurate.
The generated code's API has never really bothered me. It is flexible enough to be clever. I especially liked using proto3 for data types and then storing them in a kv store with an API like:
type WithID interface { GetId() []byte }
func Put(tx *Tx, x WithID) error { ... }
func Get(tx *Tx, id []byte) (WithId, error) { ... }
The autogenerated API is flexible enough for this sort of shenanigan, though it's not something I would recommend except to have fun.
I found that this ends up being a giant amount of useless code, and a ton of memory allocation noise, that only satisfied my desire for elegance. I've given up that approach and just use protobuf types throughout as the base type. I got sick of writing dumb conversion funcs.
It’s fairly mindless boilerplate for sure, but it does mean that when something happens that causes a change like this protobuf update, the change in your codebase is isolated just to the interface between it and your code ie your dumb conversion funcs. Otherwise you end up with the problem the original commenter had.
It’s good to isolate your dependencies within the code :)
At which point I loose all the benefits of lazy decoding that the accessor methods can provide, so I could just decode directly into a sensible struct, except you can’t with Protobuf.
Well it depends. If your data model doesn't include "this bool is optional", you can just include the bool directly in the struct and get all the memory layout advantages, and then you decide in your protobuf -> domain type conversion code whether it's an error if that field is missing or if it just defaults to 'false'. You only need to make ways for a field to be optional (such as naming it a pointer where nil represents "missing") when that actually makes sense in your data model.
I've done this, it only makes sense to me if you're trying to recycle some legacy code that's already using the domain types. Or else there's a bunch of extra conversion logic and unnecessary copying, feels like an antipattern
I mean calling it "a new API per message" is a bit of an exaggeration... the "API" per message is still the same: something with some set of attributes. It's just that those attributes are now set and accessed with getters and setters (with predictable names) rather than as struct fields. Once you know how to access fields on protobuf types in general, all message-specific info you need is which fields exist and what their types are, which was the case before too.
I can’t wait to try this new Protobuf Enterprise Edition, with its sea of getters and setters ad nauseam. /s
However I can get behind it for the lazy decoding which seems nice, though I doubt its actual usefulness for serious software (tm). As someone else already mentioned, an actual serious api (tm) will have business-scope types to uncouple the api definition from the implementation. And that’s how you keep sane as soon as you have to support multiple versions of the api.
Also, a lot of the benefits mentioned for footgun reductions smell like workarounds for the language shortcomings. Memory address comparisons, accidental pointer sharing and mutability, enums, optional handling, etc are already solved problems and where something like rust shines. (Disclaimer: I run several grpc apis written in rust in prod)
The article goes into great detail about the benefits of an opaque api vs open structs. Somewhat unintuitively open structs are not necessarily the “fastest” largely due to pointers requiring heap allocations.
Opaque APIs can also be “faster” due to lazy loading and avoiding memcpy altogether. The latter appears in libraries like flat buffers but not here IIRC.
It's true that this would perform better, and greatly reduce allocations. But:
- Messages (especially opaque ones) are not supposed to be copied. The
recommendation is to use `m := &mypb.Message{}`.
- This would make migrating to use the opaque API more difficult, if the
getters don't return the same type as the old open API fields, much more
code needs to be rewritten, or some wrapper that allocates a new slice on
every get.
- Users expect that `subm := m.GetSubMessages()[2] ;
m.SetSubMessages(append(m.GetSubMessages(), anotherm))) ; subm.SetInt(42) ;
assert(subm.GetInt() == m.GetSubMessages()[2].GetInt())`. This would not be
the case if the API returned a slice of values.
- ...
Effectively, a slice of pointers is baked into the API, and the way people use
protocol buffers in Go. For these reasons, it's not clear to me this would end
up performing better or causing less work.
If we had returned an iterator (new in Go 1.23) instead of an actual slice, then
it would've been possible to vary the underlying representation
(slice-of-pointers, slice-of-value-chunks, ...). But there are other downsides
to that too:
- Allocations when passing iterators to functions that expect a slice.
- Extra API surface for modifying the list (append, getn, len, ...).
Not that clear of a win either.
Another thing that could be considered is: when decoding, allocate a slice of
values ([]mypb.Message), *and* a slice with pointers (or do it lazily):
[]*mypb.Message. Then initialize:
for i := range valuel {
ptrl[i] = &valuel[i] // TODO: verify that this escape doesn't cause disjoint allocations.
}
That might be beneficial due to grouping allocations, and the user would be none
the wiser.
> - Messages (especially opaque ones) are not supposed to be copied.
So?
> - This would make migrating to use the opaque API more difficult
The opaque API is stupid to begin with. Now the objects are no longer threadsafe. You can't just read a message in one thread and process it in two different threads.
> Users expect
Then don't expect this. If you're breaking the API, then at least break it in a way that makes it better afterwards.
> Now the objects are no longer threadsafe. You can't just read a message in one thread and process it in two different threads.
This is not correct. The Opaque API provides the same guarantees as before, meaning you can read a message in one goroutine and then access it (but not modify it) from other goroutines concurrently.
> > - Messages (especially opaque ones) are not supposed to be copied.
> So?
If you have a []mypb.Message, and range over it in the normal way:
if _, m := range msgs {
// Use.
}
That makes a copy of the struct. This is not supported in general for the opaque API, even though it appears to work for standard use cases. The representation is meant to be opaque.
Copying makes for surprising semantics, and prevents some representation changes.
An example w.r.t. the surprising semantics:
var ms []mypb.Message = ppb.GetMessages() // A repeated submessage field
for i, m := range ms {
m.SetMyInt(i)
}
assert(ppb.GetMessages()[1].GetMyInt(1) == 1) // This would fail in general, due to SetMyInt acting on a copy.
This would not work as expected, as I highlighted in the comment. Basically, acting on value types means being very careful about identity. It makes it easy to make mistakes. I like data-driven code, but working around this (sometimes you'd want a copy, sometimes you wouldn't) would be a painful excercise.
You may have noticed that changing a heavily pointerized tree of types into value types often compiles with just a few changes, because Go automatically dereferences when needed. But it often won't work from a semantic point of view because the most intuitive way to modify such types uses copies (the range loop is a good example).
Now imagine changing the representation such that it carries a mutex, or another nocopy type. That would lead to issues unless those nocopy types would be encapsulated in a pointer. But then you get issues with initialization:
var m mypb.Message // Value type, but what about the *sync.Mutex contained deep within?
Also consider laziness
func process(lm mypb.LazyMessage) {
if lm.GetSubMessage().GetInt() != 42 {
panic("user is not enlightened")
}
}
var lm mypb.LazyMessage
process(lm) // Copy a fairly large struct.
ln.GetSubMessage().GetInt() // Does lazy unmarshaling of sub_message redundantly.
If you want to make the argument that individual messages should be pointers, but slices should still be value slices. Then I have the following for you:
ms := m.GetSubMessages() // []mypb.Message
el := &ms[0]
anotherEl := new(mypb.Message)
ms.SetSubMessages(append(ms.GetSubMessages(), anotherEl)) // Can cause reallocation, now el no longer references to m.GetSubMessages()[0]. But it no reallocation happened, it does.
In practice, value typing leads to a bunch of issues.
Since you seem so sure of your position, I'm actually curious. How would you design the API, how would you use it? Do you have any examples I can look at of this style being used in practice?
I don't think it is. Effective Go says that Go doesn't provide automatic support for getters and setters but there's nothing wrong with providing them yourself. Since in that case they are actually doing something (checking/updating the bitfield that contains the presence of each field), it makes sense to use them.
They are called `GetFoo()` instead of the idiomatic `Foo()`, but that is to ensure compatibility with the API where the fields are directly exposed as `Foo`, which also makes sense.
Why? I'm going to encourage my engineers and other teams to use it. Using this API would 100% have prevented bugs created by accessing the generated structs directly, especially in the presence of an optional value.
Graphql won the race for me. Grpc is no longer relevant. Too many hurdles, no proper to and from Web support. You have to use some 3rd party non free service.
No, they're both possible choices for your basic client-server communication layer that you build everything else on. (I mean, technically gRPC rather than protobuf, but protobuf is the biggest part of gRPC).
Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc.
Varlink (https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.