In this thread people talking about a civilization-scale collective action problem that we've never come close to solving, and telling us to use paper straws or regulate the oil industry.
The only even theoretical solutions are:
- A magically effective totalitarian state that watches everyone inside a panopticon.
- Magically undo the damage with a technological breakthrough.
Dario's essay carefully avoids its own conclusion. He argues that AI will democratize mass casualty weapons (biology especially), that human coordination at civilizational scale is impossible, and that human-run surveillance states inevitably corrupt. But he stops short of the obvious synthesis: the only survivable path is an AI-administered panopticon.
That sounds dystopian until you think it through:
- The panopticon is coming regardless. The question is who runs it.
- Human-run = corruption, abuse, "rules for thee but not for me."
- AI-run = potentially incorruptible, no ego, no blackmail, no bullshit.
- An AI doesn't need to watch you in any meaningful sense. It processes, flags genuine threats, and ignores everything else. No human ever sees your data.
- Crucially: it watches the powerful too. Politicians, corporations, billionaires finally become actually accountable.
This is the Helios ending from Deus Ex, and it's the Culture series' premise. Benevolent AI sovereignty isn't necessarily dystopia, and it might be the only path to something like Star Trek.
The reason we can't talk about this is that it's unspeakable from inside the system. Dario can't say it (he's an AI company CEO.) Politicians can't say it because it sounds insanely radical. So the discourse stays stuck on half-measures that everyone knows won't work.
I honestly believe this might be the future to work toward, because the alternatives are basically hell.
- Yes, I think people are already wary and have "AI fatigue" from all the chatbots and AI add-ons people have released over the past 2 years. It's already hard to convince people that your AI project isn't a wrapper flip.
- No, it's fundamentally useful in almost every industry, context, and environment.
Very similar to the 2000's internet on the one hand, and orders of magnitude more useful on the other. The difference that I see happening is like "AI literacy" taking over and thousands of failed startups getting immediately overrun by thousands of viable startups.
I'm using LangGraph for my app which is an AI ecommerce analyst with multiple modes (report builder, and chatbot). It consumes API data and visitor sessions to build a giant report then compress it back down to actionable insights for online store owners. The report runs for each customer once a day, queued up with BullMQ.
It's not super complex, in fact that seems to be the only way to get a more or less reliable agent right now. Keep the graph small, the prompts concise, the nodes and tools atomic in function, etc.
* Orchestrator choice and why: LangGraph because it seems the most robust and well established from my research at the time (about 6 months ago). It has decent documentation, and includes community-built graphs and nodes. People complain a lot about LangChain, but the general vibe around LangGraph is that it's a maturely designed framework.
* State and checkpointing: I'm using a memory checkpointer after every state change. Why? Reports can just re-run at negligible cost. For chats, my users' requirements just don't need persistent thread storage. Persistence is better managed through RAG entries.
* Concurrency control: I don't use parallel tool calling for most of my agents because it adds too much instability to graph execution. This is actually fine for chatbots and my app's reporting system (which doesn't need many tools), but I can see this being an issue for more complex agents.
* Autoscaling and cost: Well I use foundation models, not local ones. I swap out models for various tasks and customer subscription levels (e.g., gpt-5-nano with low reasoning effort for trial users, and gpt-5-mini for paying customers).
* Memory and retrieval: Vector DB for RAG tooling, normal DB for everything else. Sometimes I use the same Postgres database for both vector and normal data, to simplify architecture. I load raw contextual data into prompts (JSON dump). In my app's case, I use a 30-day rolling window of store data so I never keep anything longer than 30 days. I instead keep distilled information as permanent context, which I let the AI control the lifecycle of (create, update, delete).
* Observability: The only thing I would use evals for are prompts, but haven't found a good tool for that yet. I use sentiment analysis for chats the AI deems "interesting" just to see if people are complaining about something.
* Safety and isolation: For reports, I filter out PII before giving data to the AI. For chats, memory checkpointing makes threads ephemeral anyway - and I just add a rate limit + message length limit. The sentiment analysis doesn't include their original messages, only a thematic summary by the AI.
* A war story: I spent weeks trying to fine-tune a prompt for the reporting agent, in which one node was tasked with A) analyzing multiple 30-day ecommerce reports, B) generating findings, C) comparing the findings to existing insights and mutating them, and finally: D) creating short and punchy copy for new insights (title, description). I re-wrote it like 100 times, and every time I ran it it would screw up in a new way or a way that occurred 5 revisions ago. Sometimes it would work perfectly, then the next time it ran it would screw up again, with the same data and temperature set to 0.
This, honestly, is the main problem with modern AI. My solution was to decompose the node into 4 separate ones that each handle a single task - and they still manage to screw it up quite often. It's much better, but not 100% reliable.
Thanks for sharing this, truly inspiring. A few questions:
(1) What do you like the most about langgraph, have you tried platforms like autogen?
(2) Why using BullMQ with node, why not a solution like Temporal?
(3) I didn't got you usecase regarding memory check pointer? if things can re-run at negligible cost why do we need it?
(4) For sentimental analysis for chats are you using batch inferencing? Probably a loop keeping ready "interesting" chats for review
(5) this 30 days analysis is it happening parallelly or is it a sequential loop? why not using something like Airflow for this?
> The average person is well-meaning and reasonable up unto the this eerie point in their life where they feel existentially threatened and thrust on the stage of public opinion for the criticism of others.
This is how I feel for the past 8 years or so; like I've been forced to become more and more deranged, because it seems like everyone either fully supports or tacitly agrees with an insane narrative that one way or another paints me or people like me as an enemy.
I can't just take what anyone says at face value anymore, or give the benefit of the doubt. I know that as soon as they say a key word, or behave in a specific way, or even just dress in a specific way that I'm dealing with some kind of narrative that is openly hostile. It may not even be that I disagree, just that I don't want to signal myself that way. I just want to form my own opinions, but that's usually difficult and often insulting to other people. People flip like a switch as soon as they sense you're not going to fully agree with them.
The postmodern bent of our discourse is really hard to deal with because you get immediately deconstructed into one of maybe a dozen categories when you say/do anything: lib, grifter, shill, racist, snowflake, bootlicker, chud, commie, fascist, creep, etc.
I can't even cut my hair without someone categorizing me based off of it.
I mostly consume media through an RSS feed nowadays, and it hasn't helped at all, although I now don't have as much "content" to deal with emotionally.
With RSS I don't have to relitigate arguments and ideas in my own head in order to feel secure as much as before, but the way I interact with people is still deeply warped by the entire discourse.
Agreed with most of this except the last point. You are never going to make a foundational model, although you may contribute to one. Those foundational models are the product, yes, but if I could use an analogy: foundational models are like the state of the art 3D renderers in games. You still need to build the game. Some 3D renderers are used/licensed for many games.
Even the basic chat UI is a structure built around a foundational model; the model itself has no capability to maintain a chat thread. The model takes context and outputs a response, every time.
For more complex processes, you need to carefully curate what context to give the model and when. There are many applications where you can say "oh, chatgpt can analyze your business data and tell you how to optimize different processes", but good luck actually doing that. That requires complex prompts and sequences of LLM calls (or other ML models), mixed with well-defined tools that enable the AI to return a useful result.
This forms the basis of AI engineering - which is different from developing AI models - and this is what most software engineers will be doing in the next 5-10 years. This isn't some kind of hype that will die down as soon as the money gets spent, a la crypto. People will create agents that automate many processes, even within software development itself. This kind of utility is a no-brainer for anyone running a business, and hits deeply in consumer markets as well. Much of what OpenAI is currently working on is building agents around their own models to break into consumer markets.
I agree that instrumenting the model is useful in many contexts, but I don't believe it is something so unique to value Cursor such valuation, or all the attention RAG, memory, MCP get. If people say LLMs are going to be commodities (we will see) imagine the layer about RAG, tool usage, memory...
The progresses we are seeing in agents are 99% due to new LLMs being semantically more powerful.
So what is their claim then? They think that if you optimize your consumption based on "environmentally friendly" supply chains and aesthetics, even if those optimizations are actually unhealthy (ie. replacing cocoa with sunflower seeds) and/or useless, we'll solve climate change?
This is all about aesthetics for addicted consumers. What would actually be much better, all around, is if our culture started promoting buying high quality cocoa powder and making our own cookies instead of buying highly processed garbage that pretends to be good for the environment.
Things can help solve a problem without being sufficient to solve it on their own. Do you refuse to use LED lights because they won't bring your power bill down to zero?
"Police are harrassing the lower and middle class with frightening and coercive measures on behalf of the political elite. Here's why that's a good thing..."
The only even theoretical solutions are:
- A magically effective totalitarian state that watches everyone inside a panopticon.
- Magically undo the damage with a technological breakthrough.
- Immediately leave Earth.