As a Xoogler it's always seemed weird to me how Kubernetes was compared to Borg. Kubernetes covers a much larger set of things than Borg alone, and I don't necessarily think that's for the better. Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either.
Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" than Kubernetes ever did to me. I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares.
Consul is usually very reliable; when it breaks it can be very painful and mystifying. I've worked on removing it both from internal systems due to outages, and products, based on feedback.
* in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged
* in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers
* second-hand, in self-hosted installations, it caused a number of hard-to-troubleshoot outages
* something about circular dependencies and going by "wait _n_ seconds" rather than by healthiness
It was reliable enough that it could gather a really significant blast radius, and it had different gnarly failure modes, so documentation could be irrelevant from case to case.
I don't want to sound bad, but to me all of these sound like a misconfiguration and lack of understanding how consul works tbh. But i don't know the full context so eh, these things happen.
We rearchitected. At one workplace, we built and distributed our own service. At another, we shifted to semi-automated more static lists of servers for roles; those servers were much less dynamic.
There are exceptions but most of the time replacing an off the shelf std. solution with something self made looks like NIH syndrome to me.
The exceptions are the few cases where you know you will forever only need some strict set of features and your own solution can provide them by way more simple means than the in comparison "fat" off the shelf solution.
I'm not sure service discovery in a cluster is one of those cases.
Not OP, but go look at the consul documentation. In fact, just look at "Configuration" page alone: https://www.consul.io/docs/agent/options - it goes FOREVER. Maybe you don't need most of that, but whatever you do need, you're going to have to find it somewhere in there.
And yes, the error messages can be utterly cryptic.
One thing: Don't ever let junior devs try to create a cluster out of their own laptops (yes they will think it sounds like a good idea) as this will be a never-ending nightmare.
Borg is what k8s could be if there were any kind of ground rules and people were willing to call out some use cases as being legitimately stupid. Compared to k8s, Borg is a model of usability and simplicity.
I don't think that's really true at all. You can pretty easily run any damned thing in Borg. If you want it to do stuff with other Google services then you need a lot of Google code, but if you just want to run 100000 replicas of `/bin/sleep 100000`, Borg makes it easy.
Sure but none of the stuff like authentication and access to any google systems will work. Kubernetes is complex bc it can take damn near any docker container and it can run seamlessly
Go is a very hard language to work on as a team. It does not push a lot of conventions on you, and when it does, they just seem weird and anachronistic.
Even the original k8s codebase was notoriously horrible in Go, and that was "from the source".
Ironically, HashiCorp has a better Go codebase, and that's where I picked best practices from, not from Google.
The problem with Go is that the Googlers fail to challenge some of the design decisions. The giants of computer science working on it at the company cannot be questioned, and that just leads to a broken feedback process.
As someone said, Go is the new C, skipping decades of new language design concepts.
> The problem with Go is that the Googlers fail to challenge some of the design decisions. The giants of computer science working on it at the company cannot be questioned, and that just leads to a broken feedback process.
Well, they can be questioned but they're not very receptive to it, so most people don't bother. During my time at Alphabet I didn't see much use of Go anyways, other than in glue tooling (including a horrific large-scale incident due to non-obvious behaviour of Go code in such a glue tool).
The original k8s codebase was Java, then rewritten in Go by Java developers.
K8s is not a good example of Go, and it has nothing to do with the size of the project.
> Being written in a language that isn't well-suited to large projects and refactoring efforts doesn't help either.
I know that Borg was written in Java and Kubernetes in Go. Though the latter had a reputation in the beginning as a systems programming language, its purpose was actually to build large-scale cloud infrastructure projects with it and it proved formidably well suited for the task. It compiles fast, anyone can read it, good tooling and it is efficient for the layer on which it is meant to be deployed. Go, as Java is one of the most productive languages in use today, judging by the ecosystems they have spawned.
I elaborated a bit in my reply above, but basically, multiple logging frameworks with incompatible APIs, few of which offer the fine-grained control the person running the app might need.
But I'm used to the JVM world. And when I first met slf4j, I was like, wtf is this crap, but I appreciate it now as the embodiment of the desire to standardise logging across the Java ecosystem.
While using slf4j does make it trivial to swap out logging frameworks in an app, in my 13 years at my last job, we only did that once, from log4j to Logback, so that's not so important.
But yeah, what I miss from Java land in Go logging is the common approach - loggers and appenders are (usually) configured outside of code, the user can provide their own configuration at runtime to override the config shipped in the jar to troubleshoot issues - especially when you can configure the logging lib to check the conf file every X seconds for changes - allows you to change logger levels on the fly without restarting the app (ditto Logback's JMX configurator).
And lastly, no matter the logging library, configuring them is near identical.
klog is the descendant of glog which is a go implementation of Google's logging.
Go has nothing to do with it except it's the logging that k8s took on (originally it used glog).
Sorry was referring more to the many different incompatible logging libraries in play.
Also the inability in the ones I've used for the person running the app to set a particular logger to a desired level easily.
I think klog can do this with the vmodules flag, if, and only if, the devs used klog.V() for their logging statements.
Logrus requires the dev to allow the user to configure the log level, common idiom I've encountered is doing so via an env var, but IIRC, that applies to all logging in the app, no way to limit it to particular files/modules.
It's been a real pain at times for me. Feels like the Go philosophy on logging focuses more on the dev controlling it, than the person running it.
There you Go (pun intended). Go even replaced a low-level language like C++ and achieved the same result in the end. I don't know why I thought it's Java, probably the first Kube was initially in Java. It's even better that they managed to pull that off.
But it did not replace it. Google runs on Borg. Go lets you build things fast, but the lack of strong typing and the vast amount of language pitfalls make maintenance hard in the long run.
The community also has the attitude to pretend that these pitfalls don't actually exist, which is very different from C++ where most peculiar behaviours are well-understood and controlled.
Sorry, I don't think I have been explicit in my comment. The language of K8s, a system which is the so-called direct descendant of Borg, replaced C++ at that layer. Also, Go is strongly typed.
I'd argue that go is statically typed rather than strongly typed due to `interface{}`.
> There is no real agreement on what "strongly typed" means, although the most widely used definition in the professional literature is that in a "strongly typed" language, it is not possible for the programmer to work around the restrictions imposed by the type system. This term is almost always used to describe statically typed languages.
Actually Google knew at the time they "designed" k8s that Borg doesn't scale due to fundamental design flaws in it's basic architecture. Still they reused the exact same architecture for k8s.
Who wants to know the details of those scaling issues can google for the Omega paper.
The "proper"™ solution to those design flaws was implemented in Mesos (and to my knowledge nowhere else until now).
> It really is about time developers stopped pushing their own personal preferences as if it's some kind of fact.
Agreed, so let's all go to Rust or OCaml because they have very strong static typing systems. ;)
Both C++ and Go have plenty of warts, with C++ having way too many footguns and Go allowing you to use it as a dynamic language whenever you figure it's too much work to go through your problem with static types.
> Go even replaced a low-level language like C++ and achieved the same result in the end
Did it, though? Honest question. I get the feeling that it ended up competing with Java (and Python) more than it ended up replacing C++. The C++ folks seem to be way more into Rust than Go.
I'm a Java fan and an ex-Scala aficionado. I would have hoped that JVM would eat the pie when it comes to cloud deployments but it didn't happen. Like Scala.js never happened and TypeScript became the type system of the web. JVM languages will remain at the application layer, concerned with microservices, data processing, streaming, databases, etc. It's not what folks seem to be way more into, it's all about tradeoffs. I am talking here about layers and the suitability of a language at a specific layer. I don't know about Rust, but Go proved that it can handle it. If Rust would prove to be better, that would be great, but only time will tell. Until now it did not happen, instead people are trying to fit Rust in scenarios where a higher-level language would go round in circles (GUI programming, microservices, game development etc.). For Java it is too late, if GraalVM with native images and value types would have been released earlier, maybe we could say that Java could compete with Go at that layer, but it is not, the train has left the station long time ago. Only if Oracle comes out of the cave and throw a lot of money at a JVM alternative to Kubernetes, which is likely to happen in the foreseeable future, given the investments and the attention the Java platform has received recently.
Plenty of K8s operators written in Java. Sure, it's not not underpinning K8s itself, but tbh, what language K8s is written in doesn't really matter, so long as it works.
Yes, exactly my thoughts, the problem is that devops generally are used to Python/Go and in my company though they don't make it mandatory, they recommend Go. Also, they have a repulsive reaction to everything .NET and JVM :).
> Also, they have a repulsive reaction to everything .NET and JVM
It's an interesting phenomenon I observe quite commonly.
I think in the devops space they see these VM based languages as basically introducing ten redundant layers of unnecessary assumptions and complexity on top of what is already a good foundation - the Unix OS layer. They know the unix os layer well but every time they deploy one of these VM based languages it creates headaches through unique and strange behaviour that violates their assumptions and knowledge and they have no way to learn it since you need years of experience as a developer to become comfortable with it all. From a developer perspective, we see the OS as this annoying ball of complexity and want VM languages to manage that and make it go away.
So it's all about what you know better and where your comfort zone is in the end. For a long time the developers had the upper hand because the OS story was a train wreck, but containers have turned that around in recent years, so now it is not such a crazy thing to pin down the exact OS and version and entire dependency chain you are going to deploy on.
Fun fact: Python and Go are both managed languages and need some kind of VM.
I guess those people are just not educated enough. Those Ops people don't know much about programming in large usually. A lot of them for example think it's OK to write serous programs in Bash. This says it all, imho.
It's OK when someone looking after admin stuff isn't a full blown programmer. It's a different kind of job after all. But this needs to be taken into account when looking at that mentioned phenomenon.
Fun fact: There are formers devs among Ops, and sometimes vica versa.
Go binaries are statically compiled usually, no VM or managed. It's a simple language for simple solutions, which is often underrated.
Not that versed in Python, but there is CPython.
Bash is totally ok in the hands of someone who uses it for good ;) Agree it has too quirky syntax when you need complexity, so not good for large stuff.
Footguns are everywhere. You usually trade one in for another.
Go binaries still include the Go runtime which handles GCing etc., much like Python.
They're not VMs, but it's really a semantic difference, if you're comparing it to Java, especially now that GraalVM native compilation is gaining popularity.
I moved from C++ to Go. It made programming enjoyable to me again. With couple of lines of code, huge and well-thought system libraries and very accessible external modules and gomod I can create a working SW fast. No complicated configuratioms, no cmake, no autoconf, no cryptic compiler errors, no operator overload surprises, no slow template mess, no missing debugging symbols. Of course if you work on C++ rendering engine or you have a huge project already written with all the build and dependency workflow done and it is working fine then Go won't help you or solve anything for you. But I am not going to write anything in C++ ever again (except firmware for microcontrollers / Arduino). It was just too frustrating and too complicated. I can imagine most C++ developers still don't know all language features C++ offer because is simply over-complicated!
Nowadays I don't have use-cases for either, but from playing around with Nomad it felt a lot more "Borg-y" than Kubernetes ever did to me. I rarely hear anything about the Hashicorp alternative for service discovery and such though, so it would be interesting how that compares.