as someone that has complained about a lot of uber's resume driven development (see comments), I can attest that this system is
a.) necessary, and saves the company millions
b.) a simplification over the ... unusually ... architected system that came before and did not scale.
there's a lot of junk technology at many large tech companies, but scaling problems /are/ real when you have hundreds of millions of users and have demanding performance and reliability requirements. RAMEN is a differentiated and necessary part of Uber's infrastructure, even if that can't be said of a lot of uber eng blog posts.
> This guy used to run infra at uber and was incredibly salty about every single new technology. There were a lot of bad ones, but every conversation was about as constructive, free of evidence, and bitter as this blog post.
I'm going to assume you're a developer?
Infra people are usually much more apprehensive to take on new technology. Crucially I would describe classically trained sysadmins as 'pessimists to the core'. This is why there's memes of operations saying 'no'.
This is what devops was all about, the shared responsibility of it all. I'm going to assume that uber was perfect and got devops exactly right- but adding technology should in my mind always be met with the absolute most critical eye imaginable; and if he's a classically trained sysadmin then it probably comes from that place of being once bitten twice shy.
The apprehensiveness of sysadmins May have been justified in the world 5 years ago but today it sounds somewhat out of place. Note that critically evaluating new technologies is still an important skill and many infrastructure people I work with are extremely cautious about adopting new tech without spiking/getting to know it. But the sysadmins with the penchant for saying no is probably one of the reasons the devops movement actually kicked off: developers and infrastructure folks could see the immense productivity gains from being able to ship code quickly to production and got onboard the technologies that enabled this after being frustrated with the amount of time that traditional software deployment processes took.
So in today’s world, that kind of attitude is hardly productive. Skeptical? Absolutely. But open minded.
And this is why every single project out there is now a house of cards (or should I say house of YAML files) using insanely complicated technologies (like Kubernetes) with very "interesting" failure modes to say the least.
This attitude works today because of engineering-driven-development; the whole purpose of engineering is engineering and business priorities took a backseat in favor of buzzwords on the careers page and an obligatory "engineering blog" (describing how they solve self-inflicted problems), however when it comes to reliability and solving problems a large majority of projects can get away with much simpler, old-school technologies.
Almost everything in your comment is wrong. Kubernetes has enabled a whole host of observability tooling (eg opentracing), promoted a culture where app logs are easily and always accessible, enabled 0 downtime deployments for teams without dedicate infra specialists and so much more. It has made deploying reliable applications a lot more easier than ever before.
Services today scale to handle a lot more users and traffic than they did not so long ago; and these reliability guarantees are the norm rather than an exception.
Kubernetes does indeed have advantages but also brings a whole layer of complexity, overhead and moving parts. From my experience, in many cases the theoretical advantages don't end up being worth the tradeoff and/or don't even end up being implemented. Furthermore the particular things you mention (tracing, centralized logging & no-downtime deploys) can be done just as easily without Kubernetes.
I disagree about not needing dedicated infrastructure specialists. Kubernetes' complexity, learning curve and failure modes would make me uncomfortable operating without having a dedicated "devops" person (or sysadmin as we used to call them) while I am perfectly comfortable managing a few virtual machines (or even bare metal hosts) with a load-balancer in front of it. I recommend building systems in a way that can easily fit in your mind, and there's only so many abstraction layers and moving parts you can fit in there before you overload.
When it comes to scaling, not every application needs to scale and even when it needs to, it's trivial to scale stateless app servers without Kubernetes. You can scale quite far without Kubernetes, and when you're past that point you'll realize your main bottleneck is your data store and Kubernetes (or similar) can't magically solve that.
Your answer to everything I said is “actually, it’s not that hard without kubernetes”. Maybe it’s not for you. For most developers, it absolutely is. And that’s why kubernetes is popular.
Fads don’t form out of thin air. There’s always some value that they provide. To someone experienced with setting up infrastructure, the tasks may seem trivial, and the value add is low. For others who don’t, having a dead simple way of easily adding tooling around their applications is a godsend. Why is this so fucking hard for you to understand?
I am approaching this from a developer's perspective. Kubernetes and similar (even local Docker) introduces an extra layer of indirection that you often have to fight with. Sometimes it's worth it, sometimes it's not. For me, if my application is misbehaving in production, I prefer being able to just SSH into the machine and figure out what's wrong than fight with the container layer, its authentication system, command-line syntax just to obtain a shell inside the container. When I am developing locally, I prefer having all my files on the local filesystem instead of having to worry about volume redirection and "docker exec".
I am not an expert in setting up infrastructure by any means. In fact if I were I would probably use and promote these technologies. But in my opinion, adding another layer of abstraction doesn't magically solve the problems of the underlying stack (it won't protect you against obscure Linux kernel behavior, but now you have yet another moving part and potential variable which you need to account for when troubleshooting) but still gets in the way when you're trying to do something simple that doesn't even require any of the advantages the container technology is offering.
When it comes to "adding tooling around their applications", I am not sure what you mean but I will assume you refer to your previous examples, in which case I do not see how container technologies change the game at all. Tracing and centralized logging require your application to talk to a centralized log server (for logs, you can also output to stdout and have systemd/syslog collect and send them to the logging server) and container technologies don't change anything here.
I am not saying that container orchestration technologies provide no value. I am saying that they are often overkill for the task at hand and introduce extra complexity, moving parts and management overhead.
I'm an infrastructure type and everything you said rings true for me.
Kubernetes and abstractions of its ilk (shipping containers for example) have a place, but every abstraction comes with some form of trade-offs, be that performance or transparency.
Dealing with a node brown-out in kubernetes is much worse than dealing with a network, host or service outage because the troubleshooting steps involved evolve fractally.
That said, obviously there is value- but it's good to critically assess the value instead of just jumping in.
It is and it isn't. I remember the first time I thought "this software is just X that everybody forgot about, reinvented again". That was 2005. Not coincidentally that was a decade after I started working as a sysadmin.
Venkatesh Rao says that there comes a point in your life when you realize things you thought were permanent are temporary, and things you thought were temporary are permanent; he uses "40" as a good rubric for that developmental stage. IT goes through many pendula, whether it be containerization vs. amalgamation, or thin clients vs. thick. What, over time, you learn to hold on to is the tools that have lasted decades and will probably continue to. Right now the pendulum is starting to swing back towards amalgamation, and it will probably swing back towards containerization again in another decade. Whatever the tech is at the time, it can be good to reconsider whether your views will have changed not on the technology, but on the larger pendulum it's riding.
I find it hard to parse anything meaningful from your comment, sorry. I don’t mean to be rude, perhaps blunt.
Please name what infrastructure technology has lasted decades, I am curious to know. If you try hard enough yes you can see it’s all cyclical, you can say, MULTICS was the OG cloud. OK, fine, but what use is that to be as a practical software developer building things? Perhaps with age you start seeing everything as being like something you’ve already seen so it’s not that exciting anymore?
> A decade is a long time in software engineering.
Is it?
Code I wrote literally 20 years ago is still running in production, and I get paid to work on an app with plenty of code around that was first committed in 2009.
If you build a system in k8s today, and it's a success, there's good reason to believe you'll be on k8s in 2030.
That’s pretty cool, really. I can’t really find studies on average software life time, so all I really have is anecdotes. I can’t imagine software running for that long on eg public clouds without any kind of major refactor considering how quickly technology changes today, and how aggressively cloud costs push companies to change. Specifically, I can see something like “Serverless” coming to maturity and completely changing the paradigm (again!) on how software is architected. But it’s possible I’m wrong and like you say, k8s will be around for much longer than I anticipate.
I'm explaining that the majority of the people who were sysadmins in 2005 are more likely to be pessimists than optimists and are change/risk averse, I don't think that's controversial and definitely is a meme.[0]
> [thanks for] helping me empathize with toxic people.
If you dislike the article, go ahead and say why. Leaving negative personal comments about the person who wrote it approximates your own comments: unconstructive, free of evidence and bitter
I haven't even read the article and hence don't really have an opinion either way, but "[not] constructive" and "free of evidence" seem like pretty specific criticisms to me.
Given that Uber is infamous for its extreme NIH syndrome, unnecessarily reinventing the wheel and spending enormous amounts of dev effort to create internal versions of products that already exist, I think it's relevant to the article to point out that the author almost probably had a key role in that culture, and what his philosophy leads to if taken unchecked.
Its safe to say the unix crowd recreated the Windows Registry. I guess it turns out the Microsoft guys weren't idiots its just a messy problem that requires a messy solution.
etcd is not a windows registry- it's more similar to active directory, but with consensus protocols to implement the multi-machine requirements for consistency.
I appreciate so many of you are willing to crusade for free speech. But the reality is that platforming isn't free, and the process of deciding who gets platforms is political. The first amendment doesn't govern this process, nor can it—no one can reasonably consume every piece of intellectual content, from every angle, and process it.
My point is, cancel culture definitely exists. There has always been, and always will be, political processes for selecting views we consider acceptable discourse. Arguments against cancel culture that don't grapple with this reality are missing the point. Canceling isn't about using the power of the state to crack down on free speech. It's about deciding who gets access to a scarce resource: their platform.
The way this get decided is unavoidably political, since it's about who wields power.
A good example is this website. While most posts are technical, Hacker News hosts a significant amount of discussion regarding social issues. If you read this site using hckrnews.com, which preserves posts flagged off the front page, it's easy to notice that there are political trends in how moderators and users select what posts get prominence. This post, which espouses a more centrist perspective on cancel culture, was briefly flagged, then restored. Many, many other posts espousing more progressive views are systematically downvoted or flagged (search for DEAD on hckrnews.com), many of which are far longer and more carefully developed than Taibbi's piece. Regardless of how you feel about confederate statues, I think it's hard to consider that this 538 post (https://projects.fivethirtyeight.com/confederate-statues/) isn't comparable to this post in terms of quality, sophistication, and merit to be on the front page of HN.
For those of you who agree with Taibbi... do you think those posts aren't "canceled"?
If you think this example is contrived, because the nature of this cancellation doesn't involve a visible mob, that's because of power differences.
The nature of "cancellation" against minorities looks different. Mob justice isn't necessary when people in power can just fire you. And progressives are fired or otherwise censored for advocating for inclusion, against police violence, for progressive social policy all the time. You don't see mob justice in these cases because this kind of canceling is often enforced by institutions. A mob isn't necessary. The effect is the same: people are regularly censored and excluded from mainstream prominence because their views are considered unacceptable.
The reason this latest wave of (attempted) cancellations (nyt editorial, jk rowling, adam rapaport, countless people resulting from #metoo etc) have garnered attention is twofold:
1. They require popular support. Lots of people support holding rapaport, the founder of crossfit, Tucker Carlson, etc to account. No institution wanted to punish these people, so a popular movement formed.
2. They cancel views formerly considered to be within acceptable bounds. Anti-trans speech, implicitly discriminatory pay practices, military crackdowns on protestors, even if they weren't popular, are believed by many to be "newspaper publishable". It's shocking to people like Taibbi that people who share their views can get fired and singled out on social media—things that happen to people with progressive views on a regular basis.
3. Oftentimes, the people making the speech are civil and ostensibly in good faith, even if their views are onerous.
To understand cancel culture, you need to define cancellation as "people whose views are censored", and you need to look at people who espouse progressive views. That "cancel culture" is prominent today means that our definition of acceptability is changing, that powerful people require mobs to be held accountable, and that we increasingly believe that how you present the argument (good intent, good faith etc) is less important than what you present.
Lastly, I'd be remiss if I didn't acknowledge that cancellations are sometimes off the mark. Mob justice is often unfair and harmful. It is often disproportionate (the shor incident is the best example I can think of). Holding people accountable for hate speech or discriminatory practices should be the job of institutions. But they're not doing the job—this is what progressives are protesting.
If you want to stop mob culture, change institutions to fairly enforce the injustices that the mob is necessary to enforce today.
Oftentimes, these tools are developed for much more complex sites than the people who author these rants have ever maintained. Of course it doesn't solve meaningful problems for sites that can be maintained by a handful of developers and whose feature set evolves slowly.
That's fine. Don't use them.
It'd be nice if rants like this also reflect on why these frameworks do exist. Then people would actually learn something.
Here are some of the tradeoffs I considered when evaluating JAMStack vs flask/jinja server side rendering.
1. Low latency interactivity unlocks all kinds of user interaction optimizations for us. The size of the initial download is amortized over hundreds to thousands of lightening fast interactions.
2. It's difficult to manage complexity around large html/css deployments, since CSS is a global namespace. Devs use all kinds of conventions for modularizing CSS. You can definitely succeed with these strategies, but when you're juggling tens of thousands of lines of CSS, it's a lot easier to solve these problems using a real programming language and its module system. We anticipated thousands of lines of CSS over time.
3. Immutability makes whole classes of race conditions unlikely to impossible. Do things still race in React? Of course. Are they vastly less likely to happen? Yes.
4. Managing interactivity through server-client interaction shifts a ton of complexity to the backend that the browser could otherwise handle. If you're designing a multi-stage form, for example, you often have to save incomplete user submissions in servers, depending on use case. Offloading this to localstorage on the frontend is a lot easier to author and maintain.
5. If you host your frontend statically on CDNs, that's several orders of magnitude less traffic you have to scale your web servers for. The vast majority of my company's traffic is handled by Cloudfront, making our operational footprint tiny.
Are there also reasons the modern stack sucks? Yes.
1. Configuring react and webpack is incredibly, incredibly hard. Webpack exposes a vast array of knobs that most applications don't benefit from tuning.
2. Time to first load is absurdly slow and server side rendering is, again, very complex to implement. React is not the right choice for a blog or anything that is sensitive to bounce rates.
3. MUI's documentation and APIs are painful, as OP notes.
For these reasons, and because most of our application was interactive, not read-only, we went with JAMStack.
I definitely think there's a place for the traditional HTML/CSS/jQuery stack. Read-only sites, for starters, with limited feature complexity. Everything I've built in the last several years has grown in complexity and features at a faster-than-linear rate, however, and these new technologies make managing such codebases vastly simpler.
That's right[1]; and the author of "Why I'm leaving Elm" understands this, yet claims that it's not possible somehow because the Elm people are hostile to it, and that's one of the reasons the project supposedly isn't really "open source".
Something seems a bit off in the reasoning. The only reason you can't fork something is that either you don't have all the code, or there is a license problem.
One way not to have all the code is that there is a dependency on specific SaaS server installation, whose source code isn't available. If that's the case with Elm, I missed the coverage of it in the article somehow. I did get the part that the packaging ecosystem depends on a particular server controlled by the Elm project.
1. Well, not a solution for the project, but for some of its unhappy users. The project, as such, perhaps doesn't even feel that it has these problems that require solving.
He doesn't claim it's not possible, he claims that the Elm community will excommunicate you for forking.
I can't think of a language or platform that doesn't have some degree of "soft" forking that maintains communion with the language community. It's common for proprietary reasons (linux kernel, anyone?) as well as experimental reasons (e.g. PyPy). So this is an eyebrow raising claim.
What would they do? I'm not sure there's an actual threat to follow through on. They can't block a forked compiler from, for example, using official packages without close-sourcing their own compiler (to prevent it from bringing in whatever change makes it work again).
At that point, why bother respecting the threat? A forked compiler isn't going to get maintainers? A package depending on the forked compiler isn't going to be maintained? The social cost is possibly something to think about, but if one is set on leaving the community anyways then it's a choice between you choosing to not interact with the community and them (possibly!) choosing to not interact with you.
The community is more valuable than the code, because the community writes the code. So being cut off from the community is a major blow to a fork, and substantially increases their work burden.
Your fork isn't a significant amount of work, unless they've really threaded in the must-be-in-Elm-Kernel check throughout the entire compiler. Versus trying to rewrite your dependency in Elm rather than enabling importing JS, enabling JS is likely to be simpler since the functionality already exists to let Kernel modules do it.
This discussion about whether or not the "Elm community" will be meanies is missing the point and inventing a hypothetical scenario where you get kicked out of some club. Kind of a weird conjecture to me. I guarantee nobody truly cares that you fork Elm. The thing is that generally people who threaten to fork Elm are quite hostile on the Elm forums and subreddit.
If you fork Elm, an already tiny ecosystem, you'll realize that the hard part is building the community, not adding your pet features.
Everyone who has threatened to fork Elm has realized this at the end of the day. It's also why people overlook the lack of their pet features: because ecosystem is far more important.
> I guarantee nobody truly cares that you fork Elm.
One of my coworkers once edited the elm compiler to remove the native code restrictions, and placed it on NPM. Evan emailed him and asked him to take it down.
There may be more to the story that I don't know, but from what I know it sounds like Evan does care.
> I guarantee nobody truly cares that you fork Elm.
The article cites a comment by one of the elm core maintainers where that maintainer says he is opposed to the author forking Elm, and will consider it an attack on Elm's goals if he does. So I think we can safely discard this conjecture.
I don't understand your argument about ecosystem being far more important. People ceasing use of any of the tools necessarily removes them from the ecosystem and community. They have no reason to care about those things.
Definitely. At that point, they need to weigh the cost of migrating their existing project to a new platform, or to maintain the patches that will allow the old platform to have the flexibility they need.
It's important to remember that they aren't being removed from the Elm community - they can still use the packages, the compiler will still get updates they could presumably rebase their patches on.
It's still worth knowing that they're being jerks about it, since that falls outside of the social (but obviously not the legal) aspects of open source.
> falls outside of the social (but obviously not the legal) aspects of open source.
Not really, IMO. Forking a project used to be a really aggressive thing to do back before GitHub made it so common among younger waves of developers.
If you go to Evan and say, "Look, I want to use your thing but do it my way." and he says "No, I have a clear vision of what the thing is and it's not that. KTHXBYE." I don't think that's "being a jerk".
On the face of it, my impression is that excommunication, if it happens, is more likely to be caused by the author being burdensome to deal with, than code forking per se.
Of course the author is unlikely to present it that way.
And although I felt the author was moralising in this critique, it may be they are not particularly rude the rest of the time. But even writing a plethora of well-reasoned but difficult to handle posts about what's wrong with your project can be too much.
Well-established languages, platforms and projects such as Linux that have a large labour pool with socially-established patterns of working don't have the same problem, because there's enough labour to deal with it.
Smaller projects just have to turn people away when they become hard work beyond the capacity of the project's core maintainers to handle it, though.
The solution found via FLOSS is to license software so that forking is always possible when a project cannot sustain different visions for the project's direction. This diffuses the tension when people have incompatible needs from the project. Sometimes it comes with drama, particularly if people are competing for attention and trying to persuade others to follow them, but that seems inevitable because of the competition.
It is still understood that forking is permitted and intended to be part of the solution, and the social niceties are that you may be encouraged, perhaps strongly, to go away and run your own fork yourself with your own resources, under a new name/domain/etc. while acknowleging where it came from. Then it's your own job to build a reputation too; it's only fair.
It's very difficult to keep running a project while your competition lingers on the same mailing list, constantly funnelling people towards their fork in the hope of making it more popular. That's a very good reason to "excommunicate" some people, or to forbid some topics such as advertising the other project repeatedly.
> Of course the author is unlikely to present it that way.
I don't really get this accusation in light of the author leading the article with admissions of their own failings in terms of how they communicated with the maintainers of the project?
I'm writing in response to this remark, presented in the parent to my comment:
> he claims that the Elm community will excommunicate you for forking.
If that is correct, what I said is less accusation and more descriptive, and should really say something stronger: "the author hasn't presented it that way."
However if the parent to my comment is mistaken, then I agree that would make that sentence in my response unfairly speculative.
The bit about "excommunication" is very explicitly linked to the author's plan to fork Elm. Now, I don't think "excommunication" is necessarily the correct descriptor. The Elm maintainer who made the threat said that he considered making a fork on "attack" on Elm's goals [1], and that the project would not be greeted with open arms. (Although in the original comment it was a little less clear whether it was the project or the person to whom arms would not be open.) In any case, I think the comment is very clear that the hostility is to the action of forking, irrespective of previous communication problems.
I don't see anything about making a fork in the linked GitHub issue.
(The word "fork" appears once elsewhere, in an unrelated context in a different comment.)
The thing about open arms is not about forking (or if it is, it's not obvious to me), and reads to me more like "assuming you are not making your own fork, can you please stop pressuring upstream to accomodate designs which are explicitly against our clearly communicated design goals".
The quote is:
> @spookylukey It's one thing to build tooling around an implementation flaw without knowing the history, but that's all pretty well communicated at this point.
> If you understand the design goals, but don't agree with them, why not channel that in a positive way - e.g. by building something that fits your vision instead of directly working against Elm's design goals?
> As someone who has spent a lot of time collaborating with many others to help Elm achieve its stated design goals, intentionally working against those goals feels to me like an attack on our efforts. We have been really clear about our design goals in this area, and you shouldn't expect a project that works against those goals to be greeted with open arms—especially not from those of us who have been working hard for years to achieve those goals.
The comment immediately before rtfeldman's references patching the compiler for this project, which is an implied necessity if js code is to continue to be used. It's essentially a soft-fork, and that is what rtfeldman is objecting to.
Thanks, fair enough although rtfeldman appears to be responding to spookeylukey about divergent design goals, while it's norpan who is talking about maintaining a locally patched compiler.
I still see no objections to maintaining a fork or local patch from rtfeldman. Just "if you go against our explicit design goals don't expect us to want to merge it upstream for first-class support".
TBH a locally patched compiler sounds like not a huge deal to me. I have lived with locally patched GCCs before :-)
But maybe I'm unusual. I surprised someone, once, when they found some code not working and I suggested they look at their compiler source for the cause. Their response: "Wow, I hadn't ever thought of the compiler as something that might have a bug, let alone read and modify it".
IT was norpan and spookeylukey talking about patching the compiler together, and rtfeldman jumping in with references to PAST discussions of design goals with spookeylukey.
the last thing i want is to get into drama with developers of a project, so if my choices are to give up and leave or fork and face drama, then i'll give up and leave.
how that affects the Open Source status of a project is irrelevant, this is simply not a project that i could use, and thus for all intents and purposes, for me at least, it's a project that can't be forked.
If you fork quietly, you can avoid the drama. Just do your thing and don't respond to anyone (in any forum) who isn't reasonable and civil. Don't start drama, don't feed drama.
if i am happy to just quietly run my private patched version, sure. but that's not the point, because the claim is that for a good Open Source (or Free Software) project, it must be possible to go public, and once you do that, if the original developers are unsympathetic then drama is unavoidable.
there have been hostile forks in projects before, even those that eventually had a good ending. egcs for example.
i was part of such a forked project too. it wasn't meant to be hostile, and the drama was limited to the project leaders, but it happened, and feelings got hurt.
You dont even have to push anything forward. Example Redhat's compilation of code or "forked" by CentOS, who were providing just a little more freedom. We all know how that ended.
Just keep in sync with the main project, and keep the annoying/proprietary stuff out.
> You don't get to stay in the club.
Or you become the club. I think LibreOffice has more club going than Oracle's OOo.
I’ve never used Elm, but that’s not how reasonable open source projects work. Red Hat maintains several rather divergent forks of Linux and they’re still in the club. I personally run a fork of Linux that I maintain, and I’m in the club. The only people who attract serious ire from the club are people who distribute out-of-tree modules that play poorly with the rest of the system. Even in that case, no one gets excommunicated, but the upstream kernel makes no particular effort to keep problematic modules working.
The open-source ethos means welcoming the friendly competition that comes from a fork. Look at the grandparent's link for the attitude taken by awk and bash.
Get off your high horse. It doesn't matter what it was originally intended for if hospitals are over capacity, and opening up the ship will save hundreds, maybe thousands.
I think you make some good points here, and RLS is an ideal data security model in many respects.
The parent's point is that tooling around this is weak, which undermines its value. Not being able to review and easily test authz changes will almost certainly lead to issues.
> Delegate authorization logic to the business logic layer
If you take security or performance debugging seriously, you should never expose database models through APIs directly in a production app.
To illustrate, say you have an Employee model:
query {
employee(userId=uuid) {
name
salary
}
}
Say you add some hacks on top of this to only allow users to query their own employee data, believing this provides adequate security.
The next day, someone creates a Manager object, a relation from employee to Manager, and Manager to employee.
Now, without having consider security for a second, you've granted all employees the ability to query each other:
query {
employee(userId=uuid) {
name
salary
manager {
employees {
name
salary
}
}
}
}
To say that these problems occur in the wild frequently is an understatement. Since these graphql frameworks also expose introspection capabilities, discovering these exploits can be automated using crawlers. If you write a bug like this, and you will, people will find it.
Please, please stop encouraging people to directly expose their databases through an API.
Hasura avoids this issue by having it’s own row level security model. In fact they go one step further in that for any subscriptions, they can run one query for all connected clients by building a temp table with a row per client using their session variables to join against. (They poll for subscription updates since it’s fast and scales well)
Row-level security is not at all dependent upon DB users/roles. Storing app user data in a table for lookup is sufficient. Storing the user info in a session config value as provided by JWT can be even better.
Basic database table replication will suffice for the former. No replication required for the latter.
I agree that this is indeed a problem, but your proposed cure is unnecessarily onerous. At least with Hasura's equivalent product, allowing queries on related objects is an opt-in process for the admin, after each related object is defined.
I think a better piece of advice is: Please stop allowing people to query relational models automatically, and surface a separate locked-down schema for the GraphQL user.
These GraphQL over DB tools have real value. I've moved from using OpenResty to Hasura in order to surface Postgres APIs, and the time saved has been significant.
Let's be clear: many folks today are directly exposing their databases through REST. The API protocol really doesn't matter. CRUD has no affinity for any one technology or methodology.
GraphQL is no more vulnerable to crawlers than any REST server with OpenAPI on it. And yes, introspection can be disabled.
It's not like tools like Hasura, Prisma, or Postgraphile have no security baked into their products, often via a cryptographically signed JWT.
Having a JWT or whatever authn is irrelevant. The problem is an insufficient authz model.
Exposing databases through APIs is not the problem. Exposing relations without authorization is.
Under these frameworks, you can add an innocuous relationship between two models that entirely compromises security without even touching the API code. Not only that, but the graph of relations and their associated ACLs is complex. Every time you add a relation, you need to create a graph of your data model and ensure that it's safe. There are far, far more surface area to make a critical error, allowing attackers to exfiltrate large volumes of your data.
I've written these bugs in similar modeled systems (there were GQL like systems before GQL). I've fixed these bugs. I've caught these bugs in code review.
These bugs are orders of magnitude less likely to happen with a simpler authz model where you don't need to lock down every relation, just the table itself. This is why the GraphQL creators themselves encourage users to put authz at the business layer.
2 out of three examples above literally have no framework authz support. Postgraphile requires setting up row level security policies, meaning you have no control over what layer of code you want authz policies to live; they must be in the database. Even if you are ok with that sacrifice, you still have to find ways to manage this in version control and test, for which there is scarce tooling.
Hasura seems to do the right thing here, provided you opt into it. It's not clear if it allows you to easily version control or test your ACLs.
> And that's all assuming the GraphQL server has a public IP, which is far from a certainty (just like REST).
Security doesn't stop at your VPN. At several hundred engineers, organizations begin implementing internal controls.
In fact, my example was an employee comp manager :|.
> Postgraphile requires setting up row level security policies, meaning you have no control over what layer of code you want authz policies to live; they must be in the database.
I might go ahead and argue that RLS in the database is exactly where security on the data model should go. Past a certain size, many databases have more than one app talking to it. This means N servers that must adhere to the same authorization rules. Hopefully at least they're written in the same language.
Not all restrictions are data-oriented though, and on that note, I agree that the restrictions should live above the database using some schema-stitching or federation.
Version control is a matter of saving the schema changes through migrations and optionally DDL audits through event triggers. I would argue that testing/tooling as you seem to be imagining it is a moot point though. I find it much easier to define table, column, and row restrictions at the DB level and let those constraints bubble up. If a table is unavailable to a set of roles/groups, better to say so in the database so that no query at any higher level could accidentally allow access when it shouldn't have. Row-level security policies don't care if someone is executing a simple `SELECT * FROM example` or a 100-line monster. That row in isolation or as part of a gargantuan query reading/writing through a view or three doesn't matter. The reason Postgraphile relies on PostgreSQL itself for authorization is precisely because PostgreSQL does such a good job of it.
You are correct that simple GUI tools are lacking in this regard, but I'm not sure the use cases I've come across for it fit a GUI tool. For a data platform in a large aerospace company the method of allowing/restricting access was dependent on tagging (both AND and OR). Combine it with some temporal query support, and the RLS policies were essentially code, not just declarative. Then the GUI was the web manager UI (custom). Worked quite well. And while that data platform had some bugs we had to work out as all software does, we never had a problem with the data-level security. We even found some bugs at the app and UI layers precisely because some developers overlooked some corner cases.
App-level integration tests appeared more than sufficient to flush out any problems (or verify correct behavior) in the data-layer security.
Data-level restrictions and query auditing (below the level of an app or server) become even MORE relevant as employee size grows, not less.
If you read more of Ben's writing, he talks extensively about how software companies dominate market share through network effects and vertical integration.
You don't hear him talk about economies of scale because marginal costs are negligible for software companies. Besides, network effects and vertical integration are sufficiently powerful to control the market.
> In addition, new tools and new processes for software has increased the productivity times fold, which means that you need fewer developers for new software.
There are other barriers to entry besides the cost of writing software, like product, sales, operations, and most importantly, network.
However, the network effect in tech can be leapfrogged due to the zero marginal cost (as shown in this post). I.e. what network effect do you get from doing ML inference in the cloud?
The case for big tech today is still the economy of scale and not network effects (maybe facebook have those, but it exists only if the interface to facebook does not change).
The big tech players have economy of scale, due to their ability to use automation and offload the risk of managing complexity (I.e. one AWS engineer can manager 1000's of machines with AWS software).
No wonder, that the software that manages the public cloud is still closed source.
However, with Kubernetes operators, there is a way to move those capabilities into any Kubernetes cluser.
> The case for big tech today is still the economy of scale and not network effects (maybe facebook have those, but it exists only if the interface to facebook does not change).
This is only true if you believe that the greatest cost of developing software is running hardware. The greatest cost of developing software is developing software. Not only are economies of scale in compute management negligible except at massive scale, the cost of compute has declined dramatically as the companies you've described have made their datacenters available for rent through the cloud. Yet the tech giants persist.
Facebook, Google, Netflix, Amazon all have considerable network effects that you're not considering. For each of these companies, having so many customers provides benefits that accrue without diminishing returns, giving them a firm hold on market share. See https://stratechery.com/2015/aggregation-theory/
Ben is saying that the only way to topple the giants is by working around them and leveraging new computing technologies better than them. He makes the (admittedly speculative) case that this is no longer possible because we can't bring compute any closer to the user than the mobile devices.
> However, with Kubernetes operators, there is a way to move those capabilities into any Kubernetes cluser.
Kubernetes, at the scale of technologies we're discussing, is a minor optimization. Introducing k8s costs more than it helps far until far into a company's infra maturity. Even if most companies deployed k8s in a manner that significantly reduced costs, it's not enough to overcome the massive advantages existing tech companies have accrued. Not to mention all of the big tech companies have internal cluster managers of their own.
I mean, what more of network effect did MSFT had in the 90's. It was dominating both the OS layer AND the app layer (office). And yet, it does not have ANY share in mobile.
Kubernetes is not minor optimization if you think about what it is. Yes, if you see it as mere container orchestration. But it is the first time that a widely deployed, permissionless, open API platform exists.
>this is no longer possible because we can't bring compute any closer to the user than the mobile devices.
this is based on a very dubious assumption that bringing compute closer is the only path for innovation.
and even that is not true, you could imagine compute being even closer with a direct brain interface (actually you could consider google glasses to be an attempt at bringing compute closer)
a.) necessary, and saves the company millions
b.) a simplification over the ... unusually ... architected system that came before and did not scale.
there's a lot of junk technology at many large tech companies, but scaling problems /are/ real when you have hundreds of millions of users and have demanding performance and reliability requirements. RAMEN is a differentiated and necessary part of Uber's infrastructure, even if that can't be said of a lot of uber eng blog posts.