On top of that, the article says they've got settlement-free peering agreements with many top tier ISPs, which is almost a requirement for a bandwidth-intensive service like this. I'm sure it still costs them in the neighborhood of $500k/mo after all is said and done to run this backbone with all the equipment, space, power, telco fees, etc. It's rather impressive they pull this off with a staff of 32 according to TechCrunch.
I posted this comment to the article as well but I figure I might get more discussion of it here:
What exactly does "30 hours per minute of video is added each day." mean in layman's terms - 30 hours of video are uploaded each day? 30 hours of video are uploaded each minute of the day? I'm having trouble deciphering this metric.
I understood that, in total, the uploads from all the users in parallel every minute comes out to 30 hours. A bit like saying YouTube serves n hours of video every minute because there are so many videos being watched simultaneously.
I don't know what they mean by "by hand," but IMO, you should always script database changes and run the scripts in staging environments so you know you didn't forget anything and then it'll correctly run in prod.
We have a well-hacked version of ActiveRecord migrations that works with our replicated databases. We also have a number of different staging environments that we test with before deploying changes to production.
I think I've read every post on highscalability.com, and this is by far the most deep and detailed writeup they've done. Lots of insight into the huge scale of justin.tv and what it takes to push that kind of traffic.
Highlight for me was discovering twicecache. Love the narrow scope and "fit" for the job it's intended for.
Twice is great for us, but it's one of those things that evolved to meet our specific needs and architecture. If I were starting a new Rails app tomorrow, I'd start with Varnish, and only move away from that if absolutely necessary.
