The post said 450 million pageviews, likely since November. If we make very generous assumptions and assume that each pageview is a megabyte (very generous based on my own experience scanning billions of websites), then that's 450TB total in traffic. If you really did 450TB per month in traffic, you would need slightly more than one gigabit line (and hence VPS), but not more than two. With Hetzner the traffic would cost you €450 or $535.
Well, https://jmail.world/jacebook-logo.png is 670KB by itself and loaded on initial load, so I think they might have blown your suggested traffic budget and still have some optimization to do.
Because inexplicably, there's random pixel-level noise baked into the blue area. You can't see it unless you crank up contrast, but it makes the bitmap hard to compress losslessly. If you remove it using threshold blur, it doesn't change the appearance at all, but the size is down to 100 kB. Scale it down to a more reasonable size and you're down to 50 kB.
None of this is due to "modern web development". It's just about a dev not checking reasonable asset size before deploying/compiling, that has happened in web, game-dev, desktop apps, server containers, etc. etc.
This should be an SVG (a few kb after proper compression) or if properly made as a PNG it'd probably be in 20-ish kb.
The dev not having the common sense to check file size and apparently not realising that the PNG format was being grossly misused for this purpose (by not even having a single tone of white for the J and the corners, let alone for the blue background) is modern web development.
What is that noise actually? It's clearly not JPEG artifacts. Is it dithering from converting from a higher bitdepth source? There do appear to be very subtle gradients.
I would bet it's from AI upscaling. The dark edges around high contrast borders, plus the pronounced and slightly off-colour antialised edges (especially visible on the right side of the J) remind me of upscaling models.
If it's large enough for say 2x or more "retina" usage... a very minor soften filter, then color reduction to get well under 256 colors (often can do 32-64) quantization and oxipng w/ zopfli can go a long way... just getting down to a palette over rgb for png brings down sizes a lot... palette reduction to around 32 colors does a bit too. Just depends on the material.
That said, the actual size of this image is massive compared to where it needs to be, and looking at it, should definitely be able to quantize down to 32-64 colors and reduce the size to even 4x the render size... let alone just using svg, as others have mentioned.
Oh, ding ding! Opening in a hex editor, there's the string "Added imperceptible SynthID watermark" in an iTXt chunk. SynthID is apparently a watermark Google attaches to its AI-generated content. This is almost certainly the noise.
Fair enough, I just loaded some pages and some of them are even bigger than 2MB. But then again those static resources would be cached client-side. So unless you have 450 million unique visitors who only ever go to one URL on your site, you are looking at significantly less per pageview. I reloaded the frontpage with caching enabled and it was ~ 30kB of data transfer.
For high traffic check places like datapacket (no affiliation), or slightly cheaper, places like onlyservers (no paid affiliation, was a customer) or even find a transit provider and a colo and a server yourself. For $535 a month or less you can get a pretty good amount of bandwidth.
Genuine question: How is that a value proposition when Cloudflare offers a CDN for free with unlimited bandwidth, that you could just put in front of the sweaty VM in Helsinki?
Not trying to be obtuse, I really don't get how other providers can compete with that, I can't imagine Vercel's CDN is so significantly superior to make it worth it.
For that matter, the entire site could be in a Cloudflare worker with all the content in R2 (no egress fees, just storage). Likely to barely exceed the $5 baseline price for a paid account. (not sure on the storage costs, but likely well under $100/mo)
Yes, and I didn't mean to imply that a single VPS is all you needed. But I wanted to put things into perspective for the other posters who claimed that you couldn't possibly serve a site like this from a single machine, purely in terms of performance.
Thank you for sharing, this looks really cool. The simplicity of setting this up and operating it reminds me a lot of nsq which received a lot less publicity than it should have.
That’s a great comparison nsq is a project I have a lot of respect for.
I think there’s a similar philosophy around simplicity and operator experience.
Where Ayder diverges is in durability and recovery semantics nsq intentionally
trades some of that off to stay lightweight.
The goal here is to keep the “easy to run” feeling, but with stronger guarantees
around crash recovery and replication.
I still don't get how folks can hype Postgres with every second post on HN, yet there is no simple batteries-included way to run a HA Postgres cluster with automatic failover like you can do with MongoDB. I'm genuinely curious how people deal with this in production when they're self-hosting.
It's largely cultural. In the SQL world, people are used to accepting the absence of real HA (resilience to failure, where transactions continue without interruption) and instead rely on fast DR (stop the service, recover, check for data loss, start the service). In practice, this means that all connections are rolled back, clients must reconnect to a replica known to be in synchronous commit, and everything restarts with a cold cache.
Yet they still call it HA because there's nothing else.
Even a planned shutdown of the primary to patch the OS results in downtime, as all connections are terminated. The situation is even worse for major database upgrades: stop the application, upgrade the database, deploy a new release of the app because some features are not compatible between versions, test, re-analyze the tables, reopen the database, and only then can users resume work.
Everything in SQL/RDBMS was thought for a single-node instance, not including replicas. It's not HA because there can be only one read-write instance at a time. They even claim to be more ACID than MongoDB, but the ACID properties are guaranteed only on a single node.
One exception is Oracle RAC, but PostgreSQL has nothing like that. Some forks, like YugabyteDB, provide real HA with most PostgreSQL features.
About the hype: many applications that run on PostgreSQL accept hours of downtime, planned or unplanned. Those who run larger, more critical applications on PostgreSQL are big companies with many expert DBAs who can handle the complexity of database automation. And use logical replication for upgrades. But no solution offers both low operational complexity and high availability that can be comparable to MongoDB
Beyond the hype, the PostgreSQL community is aware of the lack of "batteries-included" HA. This discussion on the idea of a Built-in Raft replication mentions MongoDB as:
>> "God Send". Everything just worked. Replication was as reliable as one could imagine. It outlives several hardware incidents without manual intervention. It allowed cluster maintenance (software and hardware upgrades) without application downtime. I really dream PostgreSQL will be as reliable as MongoDB without need of external services.
Postgres violated serializability on a single node for a considerable amount of time [1] and used fsync incorrectly for 20 years [2]. I personally witnessed lost data on Postgres because of the fsync issue.
Database engineering is very hard. MongoDB has had both poor defaults as well as bugs in the past. It will certainly have durability bugs in the future, just like Postgres and all other serious databases. I'm not sure that Postgres' durability stacks up especially well with modern MongoDB.
CloudNativePG is automation around PostgreSQL, not "batteries included", and not the idea of Kubernetes where pods can die or spawn without impacting the availability. Unfortunately, naming it Cloud Native doesn't transform a monolithic database to an elastic cluster
We’ve recently had a disk failure in the primary and CloudNativePG promoted another to be primary but it wasn’t zero downtime. During transition, several queries failed. So something like pgBouncer together with transactional queries (no prepared statements) is still needed which has performance penalty.
> So something like pgBouncer together with transactional queries
FYI - it's already supported by cloudnativepg [1]
I was playing with this operator recently and I'm truly impressed - it's a piece of art when it comes to postgres automation; alongside with barman [2] it does everything I need and more
I use Patroni for that in a k8s environment (although it works anywhere). I get an off-the-shelf declarative deployment of an HA postgres cluster with automatic failover with a little boiler-plate YAML.
Patroni has been around for awhile. The database-as-a-service team where I work uses it under the hood. I used it to build database-as-a-service functionality on the infra platform team I was at prior to that.
It's basially push-button production PG.
There's at least one decent operator framework leveraging it, if that's your jam. I've been living and dying by self-hosting everything with k8s operators for about 6-7 years now.
We use patroni and run it outside of k8s on prem, no issues in 6 or 7 years. Just upgraded from pg 12 to 17 with basically no down time without issue either.
Yo I'm curious if you have any pointers on how you went about this to share? Did you use their provided upgrade script or did you instrument the upgrade yourself "out of band"? rsync?
Currently scratching my head on what the appropriate upgrade procedure is for a non-k8s/operator spilo/patroni cluster for minimal downtime and risk. The script doesn't seem to work for this setup, erroring on mismatching PG_VERSION when attempting. If you don't mind sharing it would be very appreciated.
Going to have some more figuring out what's up with spilo - turns out that running that outside of k8s is rare and not much documented. But it's still patroni so this is very helpful.
Yeah I'm also wondering that. I'm looking for self-host PostgreSQL after Cockroach changed their free tier license but found the HA part of PostgreSQL is really lacking. I tested Patroni which seems to be a popular choice but found some pretty critical problems (https://www.binwang.me/2024-12-02-PostgreSQL-High-Availabili...). I tried to explore some other solutions, but found out the lack of a high level design really makes the HA for PostgreSQL really hard if not impossible. For example, without the necessary information in WAL, it's hard to enforce primary node even with an external Raft/Paxos coordinator. I wrote some of them down in this blog (https://www.binwang.me/2025-08-13-Why-Consensus-Shortcuts-Fa...) especially in the section "Highly Available PostgreSQL Cluster" and "Quorum".
My theory of why Postgres is still getting the hype is either people don't know the problem, or it's acceptable on some level. I've worked in a team that maintains the in house database cluster (even though we were using MySQL instead of PostgreSQL) and the HA story was pretty bad. But there were engineers manually recover the data lost and resolve data conflicts, either from the recovery of incident or from customer tickets. So I guess that's one way of doing business.
This is my gripe with Postgres as well. Every time I see comments extolling the greatness of Postgres, I can't help but think "ah, that's a user, not a system administrator" and I think that's a completely fair judgement. Postgres is pretty great if you don't have to take care of it.
I manage Postgresql and the thing I really love about it is that there's not much no manage. It just works. Even setting up streaming replication is really easy.
> IMO Maria has fallen behind MySQL. I wouldn't chose it for anything my income depends on.
Can you give any details on that?
I switched to MariaDB back in the day for my personal projects because (so far as I could tell) it was being updated more regularly, and it was more fully open source. (I don't recall offhand at this point whether MySQL switched to a fully paid model, or just less-open.)
SKIP LOCKED was added in 10.6 (~2021), years after MySQL had it (~2017). My company was using MariaDB around the time and was trailing a version or two and it made implementing a queue very painful.
it's easy to through names out like this (pgbackrest is also useful...) but getting them setup properly in a production environment is not at all straightforward, which I think is the point.
…in which case, you should probably use a hosted offering that takes care of those things for you. RDS Aurora (Serverless or not), Neon, and many other services offer those properties without any additional setup. They charge a premium for them, however.
It’s not like Mongo gives you those properties for free either. Replication/clustering related data loss is still incredibly common precisely because mongo makes it seem like all that stuff is handled automatically at setup when in reality it requires plenty of manual tuning or extra software in order to provide the guarantees everyone thinks it does.
Properly measuring "GPU load" is something I've been wondering about, as an architect who's had to deploy ML/DL models but is still relatively new at it. With CPU workloads you can generally tell from %CPU, %Mem and IOs how much load your system is under. But with GPU I'm not sure how you can tell, other than by just measuring your model execution times. I find it makes it hard to get an idea whether upgrading to a stronger GPU would help and by how much. Are there established ways of doing this?
For kernel-level performance tuning you can use the occupancy calculator as pointed out by jplusqualt or you can profile your kernel with Nsight compute which will give you a ton of info.
But for model-wide performance, you basically have to come up with your own calculation to estimate the FLOPs required by your model and based on that figure out how well your model is maxing out the GPU capabilities (MFU/HFU).
It's harder than measuring CPU load, and depends a lot on context. For example, often 90% of a GPU's available flops are exclusively for low-precision matrix multiply-add operations. If you're doing full precision multiply-add operations at full speed, do you count that as 10% or 100% load? If you're doing lots of small operations and your warps are only 50% full, do you count that as 50% or 100% load? Unfortunately, there isn't really a shortcut to understanding how a GPU works and knowing how you're using it.
CUDA toolkit comes with an occupancy calculator that can help you determine based on your kernel launch parameters how busy your GPU will potentially be.
FYI, adding @ before a user name does nothing besides looking terrible and AFAIK dang does not get a notification when he’s mentioned. If you want to contact him, the best way is to send an email to hn@ycombinator.com .
"Utilization" tells you the percentage of your GPU's SM that currently have at least one thread assigned to them.
It does not at all take into count how much that thread is actually using the core to it's capacity.
So if e.g. your thread is locked waiting on some data from another GPU (NCCL) and actually doing nothing, it will still show 100% utilisation. A good way to realize that is when a NCCL call timeout after 30 minutes for some reason, but you can see all your GPUs (except the one that cause the failure) were at 100% util, even though they clearly did nothing but wait.
Another example are operation with low compute intensity: Say you want to add 1 to every element in a very large tensor, you effectively have to transfer every element (let's say FP8, so 1 byte) from the HBM to the l2 memory, which is very slow operation, to then simply do an add, which is extremely fast. It takes about ~1000x more time to move that byte to L2 than it takes to actually do the add, so in effect your "true" utilization is ~0.2%, but nvidia-smi (and this tool) will show 100% for the entire duration of that add.
Sadly there isn't a great general way to monitor "true" utilization during training, generally you have to come up with an estimate of how many flops your model requires per pass, look at the time it takes to do said pass, and compare the flops/sec you get to Nvidia's spec sheet. If you get around 60% of theoretical flops for a typical transformer LLM training you are basically at max utilization.
Definitely a better high level metric than nvidia-smi, and probably fine if you just want to get a very coarse idea of whether or not your are using the GPUs reasonably at all.
But when you get to the point where you care about a few percentage points of utilisation it's just not reliable enough as many things can impact energy consumption both ways. E.g. had a case were the GPU cluster we were using wasn't being cooled well enough, so you would gradually see power draw getting lower and lower as the GPUs were throttling themselves to not overheat.
You can also find cases were energy consumption is high but MFU/HFU isn't, like memory intensive workloads
Utilisation is counted by the OS, it's not exposed as a performance counter by the hardware. Thus, it's limited by the level of abstraction presented by the hardware.
It's useless on CPUs as well, just to a much much lesser extent to the point of it actually being useful.
Basically, the OS sees the CPU as being composed of multiple cores, that's the level of abstraction. Thus, the OS calculates "portion of last second where atleast one instruction was sent to this core" on each core and then reports it. The single number version is an average of each core's value.
On the other hand, the OS cannot calculate stuff inside each core - the CPU hides as part of its abstraction. That is, you cannot know "I$ utilisation", "FPU utilisation", etc,.
In the GPU, the OS doesn't even see each SM (streaming multiprocessor, loosely analogous to a cpu core). It just sees the whole GPU as one black box abstraction. Thus, it calculates utilisation as "portion of last second where atleast one kernel was executing on the whole GPU". It cannot calculate intra-GPU util at all. So one kernel executing on one SM looks the same to the OS, as that kernel executing on tens of SMs!
This is the crux of the issue.
With performance counters (perf for CPU, or nsight compute for GPU), lots of stuff visible only inside the hardware abstraction can be calculated (SM util, warp occupancy, tensor util, etc)
The question then, is why doesn't the GPU schedule stuff on each SM in the OS/driver? Instead of doing it in a microcontroller in the hardware itself on the other side of the interface?
Well, I think it's due to efficiency reasons and also for nvidia to have more freedom to change it without having compat issues due to being tied to the OS, and similar reasons. If that were the case however, then the OS could calculate util for each SM, and then average it, giving you more accurate values - the case with the kernel running on 1 SM will report a smaller util than the case with the kernel executing on 15 SMs.
IME, measuring on nsight compute causes anywhere from a 5% to 30% performance overhead, so if that's ok for you, you can enable it and get more useful measurements.
Did I get something wrong?
reply