I bought a bundle with 512GB of RAM and an older 24-core EPYC (7F72) + supermicro motherboard on ebay a bit over a year ago, it was really an amazing deal and has made for a truly nice NAS. If you're okay with stuff that's old enough that you can buy decommissioned server stuff, you can get really high-quality gear at surprisingly low prices.
Companies decommission hardware on a schedule after all, not when it stops working.
EDIT: Though looking for similar deals now, I can only find ones up to 128GB RAM and they're near twice the price I paid. I got 7F72 + motherboard + 512GB DDR4 for $1488 (uh, I swear that's what I paid, $1488.03. Didn't notice the 1488 before.) The closest I can find now is 7F72 + motherboard + 128GB DDR4 for over $2500. That's awful
I've heard it claimed that the era of being able to do this (buy slightly old used server hardware cheap on ebay) is coming to an end because, in the quest for ever more efficiency, the latest server hardware is no longer compatible with off-the-shelf power supplies etc. (there was more but that's the part that I remember) and therefore won't have any value on the second hand market.
I hope it was wrong, but it seems at least plausible to me. I'm sure that probably fixes could be made for all these issues, but the reason the current paradigm works is that, other than the motherboard and CPU, everything else you need is standard, consumer grade equipment which is therefore cheap. If you need to start buying custom (new) power supplies etc. to go along, then the price may not make as much sense anymore.
When boxes get decommissioned it's generally the entire thing. So you can pick up used power supplies as well. Or just buy new because even if it isn't ATX it's still a widely produced item that's used across multiple product lines.
The troublesome hardware is the stuff with custom backplanes and multiple daughterboards each hosting a node. Also AMD CPUs that lock themselves to a single motherboard.
The power supply incompatibility came to fruition a long time ago. Buying used Supermicro ATX motherboards to build into servers stopped being a thing for me about 1 decade ago. But used servers and desktops, even with their non-standard parts, have continued to deliver high value for me even today.
I'm curious, what is the powe draw for such a system? Of course, it heavily depends on the disks, but does it idle under 200W?
I personally feel like I will downscale my homelab hardware to reduce its power draw. My HW is rather old (and leagues below yours), more recent HW tends to be more efficient, but I have no idea how well these high end server boards can lower their idle power consumption?
That's an "if you have to ask, it's not for you" question. Also, the noise these things make... You better have a separate garage. The constraints of a data center are really far from those of a homelab.
When I was looking in October, I hadn't bought hardware for the better part of a decade, and I saw all these older posts on forums for DDR4 at $1/GB, but the lowest I could find was at least $2/GB used. These days? HAH!
If I had a decent sales channel I might be speculating on DDR4/DDR5 RAM and holding it because I expect prices to climb even higher in the coming months.
I don't remember DDR4 ever hitting $1/GB. Was I not shopping in the right places? IIRC DDR3 settled in at $1/GB quite a long time ago and recycled datacenter DDR4 was maybe ballpark $2.50/GB at some point.
AMD also has some weird cpus like the 7c13 7r13, that are way way way below their normal price bands. You don't even have to buy used to get a ridiculous systems... Until 4 months ago (RIP ram prices).
https://www.servethehome.com/amd-epyc-7c13-is-a-surprisingly...
Oh, nice! I always wanted one of those, a many-core build server running ARM would be excellent for Yocto. Anything running in quemu in the rootfs is so slow on x86 and I've seen the rootfs postprocess step take a long time.
Though... these days, getting enough RAM to support builds across 80 cores would be twice the price of the whole rest of the system I'm guessing.
Aside from the memory cost being exorbitant, 4th/5th gen ES CPUs aren’t horribly expensive for the core count you get. 8480s and 8592s have been quite accessible.
Stuffed an 8480+ ES with 192gb of memory across 8 channels and it’s actually not too bad.
Just give it a few years and you'll be able to buy the thing for a fraction of the 'current' price. By that time it will be considered to be 'slow' and 'power-hungry' and people will wonder why you're intent on running older hardware but it'll still work just fine. The DL380 G7 under the stairs here also used to cost an arm and a leg while I got it for some finger nail clippings.
I don't agree. There are a lot of inference performance improvements to make. I think the cost of inference continues to fall, and pretty much every application of AI becomes a commodity with brutal competition.
I've long thought it would be funny to do a startup where we would make accounting software that was solely a chat interface, with the only data store being a GL account list stored in context. There is probably a VC firm dumb enough to fund it.
There is not a single OpenAI model in the top 10 on openrouter's ranking page. The market is saying something about the comparative value of OpenAI.
Edit: yes, it is true that many people do integrate directly with OpenAI. That doesn't negate the fact that Openrouter users are largely not using OpenAI.
Agreed it's not really good signal (many sampling biases) but user count is not relevant, most money is from heavy API users. 900M users with free or cheap subscription are nothing compared to even 10k heavy API users.
On the other hand, big users don't use openrouter. At $work we have our own routing logic.
1. openrouter is API usage. There is obviously consumer side
2. people often use openrouter for the sole purpose of using a unified chat completions API
3. OpenAI invented chat completions; if you use openrouter for chat completions often you can just switch your endpoint URL to point to the OAI endpoint to avoid the openrouter surcharge!
4. Hence anyone with large enough volume will very likely not use openrouter for OpenAI; there is an active incentive to take the easy route of changing the endpoint URL to OAI’s
> At what point are the models going to all be "good enough", with the differentiating factor being everything else, other than model ranking?
It's already come for vast swathes of industries.
Most organizations have already been able to operationalize what are essentially GPT4 and GPT5 wrappers for standard enterprise usecases such as network security (eg. Horizon3) and internal knowledge discovery and synthesis (eg. GleanAI back in 2024-25).
I agree, and most of my peers do as well. This is why most of us shifted to funding AI Applications startups back in 2023-24. Most of these players are still in stealth or aren't household names, but neither are ServiceNow, Salesforce, Palo Alto Networks, Wiz, or Snowflake.
Foundation Models have reached a relative plateau and much of the recent hype wasn't due to enhanced model performance but smart packaging on top of existing capabilities to solve business outcomes (eg. OpenClaw, Antheopic's business suite, etc).
Most foundation model rounds are essentially growth equity rounds (not venture capital) to finance infra/DC buildouts to scale out delivery or custom ASICs to enhance operating margins.
This isn't a bad thing - it means AI in the colloquial definition has matured to the point that it has become reality.
The differentiating factor will be access to proprietary training data. Everyone can scrape the public web and use that to train an LLM. The frontier companies are spending a fortune to buy exclusive licenses to private data sources, and even hiring expert humans specifically to create new training data on priority topics.
Core Android functionality relies on eBPF in a way that PostmarketOS does not. PostmarketOS is much more of a linux distro than Android is. They are not very comparable.
Doesn't look like it has proper worktree management. UIs that abstract away worktrees are very powerful. I vibe coded my own (https://github.com/9cb14c1ec0/vibe-manager), which unfortunately doesn't have the remote component that hapi does.
"I use arbitrarily complex software that has a rapid SDLC to obfuscate the issue with the fact that we have to have military grade encryption for displaying the equivalent of a poster over the internet".
The state of our industry is such that there will be a lot of people arguing for this absurdity in the replies to me. (or I'll be flagged to death).
Package integrity makes sense, and someone will make the complicated argument that "well ackshually someone can change the download links" completely ignoring the fact that a person doing that would be quickly found out, and if it's up the chain enough then they can get a valid LE cert anyway, it's trivially easy if you are motivated enough and have access to an ASN.
Nah, you've simply never lived in a country which is afraid of its own population and does (or tried to) MITM internet traffic. Mine does both, there was a scandal several years ago:
I'll take enforced HTTPS for absolutely everything, thank you very much. Preferably with certificate pinning and similar aggressive measures to thwart any attempts to repeat this.
People will argue with you because your initial quoted sentence is chock full of fallacies.
* Caddy's complexity (especially when it comes to TLS) is not arbitrary, it's to meet the needs of auto-renewal and ... y'know, hosting sites on TLS.
* Caddy's SDLC is not, as far as I understand it, especially rapid.
* Implying that "military grade" is some level of encryption beyond the minimum level of encryption you would ever want to use is silly.
* The Manjaro website is not "the equivalent of a poster", and in fact hosts operating system downloads. Operating system integrity is kinda important.
You may have reasonable arguments for sites that are display only, do not out-link, and do not provide downloads, but this is not one of those circumstances.
Changing the links and doing nothing else would be a pretty dumb MITM. You could do a more complex variant which is not so easy to spot (targeting specific networks, injecting malware whilst modifying the checksum)
The key property of SSL that is useful for tamper resistance is that it’s hard to do silently. A random ASN doing a hijack will cause an observable BGP event and theoretically preventable via RPKI. If your ISP or similar does it, you can still detect it with CT logs.
Even the issuance is a little better, because LE will test from multiple vantage points. This doesn’t protect against an ISP interception, but it’s better than no protection.
As many others in this conversation have asked, can we have some sources on the idea that the model is spread across chips? You keep making the claim, but no one (myself included) else has any idea where that information comes from or if it is correct.
I was indeed wrong about 10 chips. I thought they would use llama 8B 16bit and a few thousand context size. It turns out, they used llama 8B 3bit with only 1k context size. That made me assume they must have chained multiple chips together since the max SRAM on TSMC n6 for reticle sized chip is only around 3GB.
Using an API key is orders of magnitude more expensive. That's the difference here. The Claude Code subscriptions are being heavily subsidized by Anthropic, which is why people want to use their subscriptions in everything else.
I think the people who use more than they pay for vastly outnumber those who pay for more than they use. It takes intention to sign up (not the default, like health care) and once you do, you quickly get in the habit of using it.
This move feels poorly timed. Their latest ad campaigns about not having ads, and the goodwill they'd earned lately in my book was just decimated by this. I'm sure I'm not the only one who's still just dipping their toes into the AI pool. And am very much a user that under utilizes what I pay for because of that. I have several clients who are scrambling to get on board with cowork. Eliminating API usage for subscription members right before a potentially large wave of turnover not only chills that motivation it signals a lack of faith in their marketing, which from my POV, put out the only AI super bowl campaign to escape virtually unscathed.
> the goodwill they'd earned lately in my book was just decimated by this
That sounds absurd to me. Committing to not building in advertising is very important and fundamental to me. Asking people who pay for a personal subscription rather than paying by the API call to use that subscription themselves sounds to me like it is. Just clarifying the social compact that was already implied.
I WANT to be able to pay a subscription price. Rather like the way I pay for my internet connectivity with a fixed monthly bill. If I had to pay per packet transmitted, I would have to stop and think about it every time I decided to download a large file or watch a movie. Sure, someone with extremely heavy usage might not be able to use a normal consumer internet subscription; but it works fine for my personal use. I like having the option for my AI usage to operate the same way.
The problem with fixed subscriptions in this model is that the service has an actual consumption cost. For something like internet service, the cost is primarily maintenance, unless the infrastructure is being expanded. But using LLMs is more like using water, where the more you use it, the greater the depletion of a resource (electricity in this case, which is likely being produced with fossil fuel which has to be sourced and transported, etc). Anthropic et al would be setting themselves up for a fall if they allow wholesale use at a fixed price.
reply