> Will LLMs be cheaper than humans once the subsidies for tokens go away? At this point we have little visibility to what the true cost of tokens is now, let alone what it will be in a few years time. It could be so cheap that we don’t care how many tokens we send to LLMs, or it could be high enough that we have to be very careful.
We do have some idea. Kimi K2 is a relatively high performing open source model. People have it running at 24 tokens/second on a pair of Mac Studios, which costs 20k. This setup requires less than a KW of power, so the $0.8-0.15 being spent there is negligible compared to a developer. This might be the cheapest setup to run locally, but it's almost certain that the cost per token is far cheaper with specialized hardware at scale.
In other words, a near-frontier model is running at a cost that a (somewhat wealthy) hobbyist can afford. And it's hard to imagine that the hardware costs don't come down quite a bit. I don't doubt that tokens are heavily subsidized but I think this might be overblown [1].
[1] training models is still extraordinarily expensive and that is certainly being subsidized, but you can amortize that cost over a lot of inference, especially once we reach a plateau for ideas and stop running training runs as frequently.
Is Kimi K2 near-frontier though? At least when run in an agent harness, and for general coding questions, it seems pretty far from it. I know what the benchmarks say, they always say it's great and close to frontier models, but is this other's impression in practice? Maybe my prompting style works best with GPT-type models, but I'm just not seeing that for the type of engineering work I do, which is fairly typical stuff.
I’ve been running K2.5 (through the API) as my daily driver for coding through Kimi Code CLI and it’s been pretty much flawless. It’s also notably cheaper and I like the option that if my vibe coded side projects became more than side projects I could run everything in house.
I’ve been pretty active in the open model space and 2 years ago you would have had to pay 20k to run models that were nowhere near as powerful. It wouldn’t surprise me if in two more years we continue to see more powerful open models on even cheaper hardware.
I agree with this statement. Kimi K2.5 is at least as good as the best closed source models today for my purposes. I've switched from Claude Code w/ Opus 4.5 to OpenCode w/ Kimi K2.5 provided by Fireworks AI. I never run into time-based limits, whereas before I was running into daily/hourly/weekly/monthly limits all the time. And I'm paying a fraction of what Anthropic was charging (from well over $100 per month to less than $50 per month).
Saw you wrote that you moved away from Opus 4.5. If you haven’t tried Opus 4.6, there’s only one number different in the name, but the common experience is it’s significantly better.
I have increasingly changed my view on LLMs and what they're good for. I still strongly believe LLMs cannot replace software engineers (they can assist yes, but software engineering requires too much 'other' stuff that LLMs really can't do), but LLMs can replace the need for software.
During the day I am working on building systems that move lots of data around where context and understanding of the business problem is everything. I largely use LLMs for assistance. This is because I need the system to be robust, scalable, maintainable by other people and adaptable to large range of future needs. LLMs will never be flawless in a meaningful sense in this space (at least in my opinion).
When I'm using Kimi I'm using it for purely vibe coded projects where I don't look at the code (and if I do I consider this a sign I'm not thinking about the problem correctly). Are these programs robust, scalable, generalizable, adaptable to future use case? No, not at all. But they don't need to be, they need to serve a single user for exactly the purpose I have. There are tasks that used to take me hours that now run in the background while I'm at work.
In this latter sense I say "flawless" because 90% of my requests solve the problem on the first pass, and the 10% of the time where there is some error, it is resolved in a single request, and I don't have to ever look at the code. For me that "don't have to look at the code" is a big part of my definition of "flawless".
Your definition of flawless is fine for you and requires a big asterix. But without being called out on it look how your message would have read for someone that's not in the known of LLM limitations, and contributed further to the dissilusionment of the field and the gaslighting that's already going on by big comapnies.
Depends what you see as flawless. From my perspective even GPT 5.2 produces mostly garbage grade code (yes it often works, but it is not suitable for anywhere near production) and takes several iterations to get it to remotely workable state.
This is what I've been increasingly understanding is the wrong way to understand how LLMs are changing things.
I fully agree that LLMs are not suitable for creating production code. But the bigger question you need to ask is 'why do we need production code?' (and to be clear, there are and always will be cases where this is true, just increasingly less of them)
The entire paradigm of modern software engineering is fairly new. I mean it wasn't until the invention of the programmable microprocessor that we even had the concept of software and that was less than 100 years ago. Even if you go back to the 80s, a lot of software doesn't need to be distributed or serve a endless variety of users. I've been reading a lot of old Common Lisp books recently and it's fascinating how often you're really programming lisp for you and your experiments. But since the advent of the web and scaling software to many users with diverse needs we've increasingly needed to maintain systems that have all the assumed properties of "production" software.
Scalable, robust, adaptable software is only a requirement because it was previously infeasible for individuals to build non-trivial systems for solving any more than a one or two personal problems. Even software engineers couldn't write their own text editor and still have enough time to also write software.
All of the standard requirements of good software exist for reasons that are increasingly becoming less relevant. You shouldn't rely on agents/LLMs to write production code, but you also should increasingly question "do I need production code?"
> Scalable, robust, adaptable software is only a requirement because it was previously infeasible for individuals to build non-trivial systems for solving any more than a one or two personal problems. Even software engineers couldn't write their own text editor and still have enough time to also write software.
That's a wild assumption. I personally know engineers who _alone_ wrote things like compilers, emulators, editors, complex games and management systems for factories, robots. That was before internet was widely available and they had to use physical books to learn.
In my mind, "yolo ai" application (throwaway code on one hand, unrestrained assistants on the other) -
is a little like better spreadsheets and smart documents were in the 90s; just run macros! Everywhere! No need for developers - just Word an macros!
Then came macro viri - and practically - everyone cut back hard on distributing code via Word and Excel (in favour of web apps and we got the dot.com bubble).
regardless its been 3 years since the release of chatgpt. literally 3. imagine in just 5 more years how much low hanging (or even big breakthroughs) will get into the pricing, things like quantization, etc. no doubt in my mind the question of "price per token" will head towards 0
You don't even need to go this expensive. An AMD Ryzen Strix Halo (AI Max+ 395) machine with 128 GiB of unified RAM will set you back about $2500 these days. I can get about 20 tokens/s on Qwen3 Coder Next at an 8 bit quant, or 17 tokens per second on Minimax M2.5 at a 3 bit quant.
Now, these models are a bit weaker, but they're in the realm of Claude Sonnet to Claude Opus 4. 6-12 months behind SOTA on something that's well within a personal hobby budget.
I was testing the 4-bit Qwen3 Coder Next on my 395+ board last night. IIRC it was maintaining around 30 tokens a second even with a large context window.
I haven't tried Minimax M2.5 yet. How do its capabilities compare to Qwen3 Coder Next in your testing?
I'm working on getting a good agentic coding workflow going with OpenCode and I had some issues with the Qwen model getting stuck in a tool calling loop.
Minimax passed this test, which even some SOTA models don't pass. But I haven't tried any agentic coding yet.
I wasn't able to allocate the full context length for Minimax with my current setup, I'm going to try quantizing the KV cache to see if I can fit the full context length into the RAM I've allocated to the GPU. Even at a 3 bit quant MiniMax is pretty heavy. Need to find a big enough context window, otherwise it'll be less useful for agentic coding. With Qwen3 Coder Next, I can use the full context window.
Yeah, I've also seen the occasional tool call looping in Qwen3 Coder Next, that seems to be an easy failure mode for that model to hit.
OK, with MiniMax M2.5 UD-Q3_K_XL (101 GiB), I can't really seem to fit the full context in even at smaller quants. Going up much above 64k tokens, I start to get OOM errors when running Firefox and Zed alongside the model, or just failure to allocate the buffers, even going down to 4 bit KV cache quants (oddly, 8 bit worked better than 4 or 5 bit, but I still ran into OOM errors).
I might be able to squeeze a bit more out if I were running fully headless with my development on another machine, but I'm running everything on a single laptop.
So looks like for my setup, 64k context with an 8 bit quant is about as good as I can do, and I need to drop down to a smaller model like Qwen3 Coder Next or GPT-OSS 120B if I want to be able to use longer contexts.
It is crazy to me that it is that slow, 4 bit quants don't lose much with Qwen3 coder next and unsloth/Qwen3-Coder-Next-UD-Q4_K_XL gets 32 tps with a 3090 (24gb) as a VM with 256k context size with llama.cpp
Same with unsloth/gpt-oss-120b-GGUF:F16 gets 25 tps and gpt-oss20b gets 195 tps!!!
The advantage is that you can use the APU for booting, and pass through the GPU to a VM, and have nice safer VMs for agents at the same time while using DDR4 IMHO.
Yeah, this is an AMD laptop integrated GPU, not a discrete NVIDIA GPU on a desktop. Also, I haven't really done much to try tweaking performance, this is just the first setup I've gotten that works.
The memory bandwidth of the Laptop CPU is better for fine tuning, but MoE really works well for inference.
I won’t use a public model for my secret sauce, no reason to help the foundation models on my secret sauce.
Even an old 1080ti works well for FIM for IDEs.
IMHO the above setup works well for boilerplate and even the sota models fail for the domain specific portions.
While I lucked out and foresaw the huge price increases, you can still find some good deals. Old gaming computers work pretty well, especially if you have Claude code locally churn on the boring parts while you work on the hard parts.
Yeah, I have a lot of problems with the idea of handing our ability to write code over to a few big Silicon Valley companies, and also have privacy concerns, environmental concerns, etc, so I've refused to touch any agentic coding until I could run open weights models locally.
I'm still not sold on the idea, but this allows me to experiment with it fully locally, without paying rent to some companies I find quite questionable, and I can know exactly how much power I'm drawing and the money is already spent, I'm not spendding hundreds a month on a subscription.
And yes, the Strix Halo isn't the only way to run models locally for a relatively affordable price; it's just the one I happened to pick, mostly because I already needed a new laptop, and that 128 GiB of unified RAM is pretty nice even when I'm not using most of it for a model.
I'm running Fedora Silverblue as my host OS, this is the kernel:
$ uname -a
Linux fedora 6.18.9-200.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 6 21:43:09 UTC 2026 x86_64 GNU/Linux
You also need to set a few kernel command line paramters to set it up to allow it to use most of your memory as graphics memory, I have the following in my kernel command line, those are each 110 GiB expressed in number of pages (I figure leaving 18 GiB or so for CPU memory is probably a good idea):
Then I'm running llama.cpp in the official llama.cpp Docker containers. The Vulkan one works out of the box. I had to build the container myself for ROCm, the llama.cpp container has ROCm 7.0 but I need 7.2 to be compatible with my kernel. I haven't actually compared the speed directly between Vulkan and ROCm yet, I'm pretty much at the point where I've just gotten everything working.
(as mentioned, still just getting this set up so I've been moving around between using `-hf` to pull directly from HuggingFace vs. using `uvx hf download` in advance, sorry that these commands are a bit messy, the problem with using `-hf` in llama.cpp is that you'll sometimes get surprise updates where it has to download many gigabytes before starting up)
20k for such a setup for a hobbyist? You can leave the somewhat away and go into sub 1% region globally. A kw of power is still 2k/year at least for me, not that I expect it will run continuously but still not negligible if you can do with 100-200 a year on cheap subscriptions.
There are plenty of normal people with hobbies that cost much more. Off the top of my head, recreational vehicles like racecars and motorcycles, but im sure there are others.
You might be correct when you say the global 1%, but that's still 83 million people.
Reminder to others that $20k is the one time startup cost, and is amortized perhaps 2-4k/year (plus power). That is in the realm of a mere gym membership around me for a family
So 5-10 years to amortize the cost. You could get 10 years of Claude Max and your $20k could stay in the bank in case the robots steal your job or you need to take an ambulance ride in the US.
Most execs I've worked with couldn't tell their engineering team what they wanted with any specificity. That won't magically get any better when they talk to an LLM.
If you can't write requirements an engineering team can use, you won't be able to write requirements for the robots either.
You can rent computer from someone else to majorly reduce the spend. If you just pay for tokens it will be cheaper than buying the entire computer outright.
Up front, yeah. But people with hobbies on the more expensive end can definitely put out 4k a year. Im thinking like people who have a workshop and like to buy new tools and start projects.
Horrific comparison point. LLM inference is way more expensive locally for single users than running batch inference at scale in a datacenter on actual GPUs/TPUs.
Why do you people trust what he has to say? Like omg dude. These folks play with numbers all the time to suit their narrative. They are not independently audited. What do you think scares them about going public? Things like this. They cannot massage the numbers the same way they do in the private market.
The problem is that OpenClaw is kind of like a self driving car that works 90% of the time. As we have seen, that last 10% (and billions of dollars) is the difference between Waymo today and prototypes 10 years ago.
Being Apple is just a structural disadvantage. Everyone knows that open claw is not secure, and it’s not like I blame the solo developer. He is just trying to get a new tool to market. But imagine that this got deployed by Apple and now all of your friends, parents and grandparents have it and implicitly trust it because Apple released it. Having it occasionally drain some bank accounts isn’t going to cut it.
This is not to say Apple isn’t behind. But OpenClaw is doing stuff that even the AI labs aren’t comfortable touching yet.
I sincerely suspect the BBC would only ever use "fired"/"firings" if the employees were being dismissed for conduct reasons, since that's the common usage in British English. I've been let go -- indeed, I've lost my job (it's the employees who suffer job losses, not the employer) -- but I've never been fired.
Which, at least in American English, comes across like corporate jargon/weasel words. Lost their job is literally true and would probably take a bunch more words to describe the precise reasons.
Both things can be literally true. I've lost my job by being made redundant, twice. In Britain redundancy is a very specific thing, where your role no longer exists and you must be let go in a fair way according to employment law. It's quite the opposite of jargon or weasel words here: https://www.gov.uk/redundancy-your-rights
I think we may be hitting an issue in translation between English and American; in British English "fired" implies "for cause", while a "blameless" process of headcount reduction is known as "redundancy". "Job losses" is a perfectly reasonable neutral phrase here. Indeed, under UK law and job contracts you generally cannot just chuck someone out of their job without either notice or cause or, for large companies, a statutory redundancy process.
People like to make too much out of active/passive word choices. Granted it can get very propagandistic ("officer-involved shooting"), but if you try to make people say "unhoused" instead of "homeless" everyone just back-translates it in their head.
> Indeed, under UK law and job contracts you generally cannot just chuck someone out of their job without either notice or cause or, for large companies, a statutory redundancy process.
This is only true when an employee has worked for a company for 2 or more years
I think American English is the same colloquially. “I got fired” means I didn’t perform or did something wrong. “I got laid off” is our “I was made redundant”.
“Fired” is also a technical term for both cases, in academic/economist speak.
> in British English "fired" implies "for cause", while a "blameless" process of headcount reduction is known as "redundancy"
OK. I was fired for no stated cause in a process that didn't involve headcount reduction, or the firing of anyone except me specifically. (The unstated cause seems to have been that I had been offered a perk by the manager who hired me that the new manager didn't want to honor after the original guy was promoted.)
And by applying these organizational changes, each person can become more load bearing and have so much more scope and impact. This is not a loss, it's a great win for everyone! /s
> Cost: $1,000
Case 1 (90%): OpenAI goes bankrupt. Return: $0
Case 2 (9%): OpenAI becomes a big successful company and goes 10x. Return: $1,000 + 5% interest = $1,050
Case 3 (1%): OpenAI becomes the big new thing and goes 100x. Return: $1,000 + 5% interest = $1,050
The actual math is that if OpenAI succeeds, then there's a nod and a wink that JPM will land the lead role in the IPO or any mergers/acquisitions, which translates into huge fees.
a company with 800 million weekly active users, and only losing $10B-$15B before implementing ads - which IMO is coming fast and soon to the LLM world - i would never calculate a 90% chance their shares end up at $0 before an exit option
This is the easiest money and best relationship JPM could imagine
Yahoo is a disingenuous parallel here. Yahoo lost because they didn't correctly embrace their market position in what's otherwise the very ripe industry of search engines. Search engines created the 4th most valuable company in the world (Google).
We don't know how ripe OpenAI's industry or market position is, yet. Yahoo knew what they had lost pretty early onto its spiral.
Also, if OpenAI goes bankrupt, you _much_ prefer to have loaned them money to having bought shares in the company. People who own shares in a bankruptcy only recover anything after all the people that loaned them money are paid back in full.
so if i'm reading this correctly, it's essentially prompt engineering here and there's no guarantee for the output. Why not enforce a guaranteed output structure by restricting the allowed logits at each step (e.g. what outlines library does)?
So in short there's no guarantee for any output from any LLM whether its Gemma or any other (ignoring some details like setting a random seed or parameters like temperature to 0). Like you mentioned though libraries like outlines can constrain the output, whereas hosted models often already include this in their API, but they can do so because its a model + some server side code.
With Gemma, or any open model, you can use the open libraries in conjunction to get what you want. Some inference frameworks like Ollama include structured output as part of their functionality.
But you mentioned all of this already in your question so I feel like I'm missing something. Let me know!
But I think you already mentioned all this in your response so I might be missing the question?
With OpenAI models, my understanding is that token output is restricted so that each next token must conform to the specified grammar (ie json schema) so you’re guaranteed to get either a function call or an error.
Edit: per simonw’s sibling comment, ollama also has this feature.
Ah, There's a distinction here with model vs model framework. The ollama inference framework supports token output restriction. Gemma in AI Studio also does, as does Gemini, there's a toggle in the right hand panel, but that's because both those models are being served in the API where the functionality is present in the server.
The Gemma model by itself does not though, nor does any "raw" model, but many open libraries exist for you to plug into whatever local framework you decide to use.
If you run Gemma via Ollama (as recommended in the Gemma docs) you get exactly that feature, because Ollama provides that for any model that they run for you: https://ollama.com/blog/structured-outputs
Under the hood, it is using the llama.cpp grammars mechanism that restricts allowed logits at each step, similar to Outlines.
I've been working on tool calling in llama.cpp for Phi-4 and have a client that can switch between local models and remote for agentic work/search/etc., I learned a lot about this situation recently:
- We can constrain the output of a JSON grammar (old school llama.cpp)
- We can format inputs to make sure it matches the model format.
To OP's question, specifying a format in the model unlocks training the model specifically had on functions calling: what I sometimes call an "agentic loop", i.e. we're dramatically increasing the odds we're singing in the right tune for the model to do the right thing in this situation.
Do you have thoughts on the code-style agents recommended by huggingface? The pitch for them is compelling, since structuring complex tasks in code is something very natural for LLMs. But then, I don’t see as much about this approach outside of HF.
Why is there a 5 year gap on your resume? It sounds like you didn't sit around twiddling your thumbs... you built stuff during that time. On your resume, treat it like you were working a job and talk about what you built. Highlight your open source contributions and if possible, tie them to your resume. Sure, some people will treat an ex-founder as a negative, but many will see it positive. You only need one job.
There is definitely ageism in tech, but 39 isn't old. I'd be happy to take a look at your resume and provide advice if you give me a way to contact you. But it sounds like the issue isn't your resume... indeed you are getting lots of interviews, so maybe it's something you are doing in the interview process. Do you have a sense of where things go wrong? Are you often getting to the final stage before hearing no?
- ~20% rejection after the technical screen (i.e., leetcoding)
- ~10% after a full round of interviews
Presumably I do or say something in interviews that is working against me. I also think that my background gives people the expectation that I should be able to reach a very high bar, and perhaps I set their expectations much higher than I can actually achieve (at least in interviews). Given my real-life interactions, I can confidently say I'm a fairly gregarious and affable person, so I doubt it's my personality that's the issue. But who knows, perhaps my ego is the problem.
Well, they certainly _might_ be functioning in ten years from now. Conservatively, you get 5 years of use out of this, which isn't bad for $15-20, depending on your use case.
Obviously in practice, this isn't always true, but in general, each employee should output more than their pay, or they wouldn't be there. As an example, Exxon Mobile supposedly has profits of $899,000/employee [1]. Their average pay is probably significantly lower than that, but let's say it's $300k, so a 20% boost in productivity increases profit per employee by $180k. An increase of 50% in salary (and I don't think it takes that much to get people to work in an office) costs them $150k.
I think that software engineering is about two things: building things the right way and building the right things.
The second one is more important than the first one. If you don't build the right product, it doesn't matter how well it scales or how it has amazing test coverage or wonderful documentation. To that end, I think that too many managers (and companies) do too much shielding of engineers from customers. If you are just given a figma mockup and told "build this", it's easy to get bogged down for a week with the details of building a search bar at the bottom of the page only to realize that the stakeholders would have been OK with a dropdown select. Better to understand the problem you are solving and the only way to really do this is to have some kind of interaction with customers. As an engineering manager, I try to encourage engineers to get on sales calls and see product demos. When you see it from a high level, you a) almost always notice things that need fixing or can be improved and b) see where the piece that you are working on fits into the larger picture.
That said, I find that many engineers don't want to get on customer calls, and usually there's room for those engineers in an organization as well. For example, "build a new video conferencing service for artists to collaborate" would be a very challenging problem (I think) that is not well defined and therefore requires deep customer understanding. "Make Google searches run with 10% fewer CPU milliseconds" is arguably a much harder problem to solve, but it's so well defined that it really doesn't need customer understanding (setting aside the initial decision about whether it makes sense to work on).
As a fellow engineering manager -- I 100% agree. The more your engineers know as much as possible about the customers, the more they will code in the right direction just be understanding.
You are given a figma that wasn't already researched and validated against requirements? If it takes a week for a team to fiddle around with a design asset only to learn customers/clients would be fine with a simpler approach, everyone failed the assignment. This was intended as a rhetorical question... I know many teams let designers waste tons of time in a vacuum and PMs are off in lalaland focused on the wrong activities, when they should be focused "building the right thing" and carefully validating that and communicating outcomes to the team and customers. Im all for letting Engineers along for the ride, but too often they (more the jr mid-level ones) are checked out during that process, not asking implementation questions or contributing to research process, until its all hindsight.
I disagree, it's great practice and starting early in a career will only pay dividends later on. It requires the desire to actually want to participate and contribute professionally and not just hide behind a keyboard all day. I get that some people just want to hide behind a keyboard all day- fine. These people often complain the most later on... in my experience.
Only a small fraction of all future AI projects have even gotten started. So they aren't only fighting over what's out there now, they're fighting over what will emerge.
This is true, and yet, many orgs who have experimented with OpenAI and are likely to return to them when a project "becomes real". When you google around online for how to do XYZ thing using LLMs, OpenAI is usually in whatever web results you read. Other models and APIs are also now using OpenAI's API format since it's the apparent winner. And for anyone who's already sent out subprocessor notifications with them as a vendor, they're locked in.
This isn't to say it's only going to be an OpenAI market. Enterprise worlds move differently, such as those in G Cloud who will buy a few million $$ of Vertex expecting to "figure out that gemini stuff later". In that sense, Google has a moat with those slices of their customers.
But I believe that when people think OpenAI has no moat because "the models will be a commodity", I think that's (a) some wishful thinking about the models and (b) doesn't consider the sociological factors that matter a lot more than how powerful a model is or where it runs.
We do have some idea. Kimi K2 is a relatively high performing open source model. People have it running at 24 tokens/second on a pair of Mac Studios, which costs 20k. This setup requires less than a KW of power, so the $0.8-0.15 being spent there is negligible compared to a developer. This might be the cheapest setup to run locally, but it's almost certain that the cost per token is far cheaper with specialized hardware at scale.
In other words, a near-frontier model is running at a cost that a (somewhat wealthy) hobbyist can afford. And it's hard to imagine that the hardware costs don't come down quite a bit. I don't doubt that tokens are heavily subsidized but I think this might be overblown [1].
[1] training models is still extraordinarily expensive and that is certainly being subsidized, but you can amortize that cost over a lot of inference, especially once we reach a plateau for ideas and stop running training runs as frequently.
reply