From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running.
The biggest thing that brings down a site is changes. Typically code changes, but also schema/data changes, infra/network/config changes, etc. As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine. The trick is to design it to be as immutable and simple as possible.
There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site.
The last thing off the top of my head that will absolutely bring a site down over time, is expired certs. If, for any reason at all, a cert fails to be regenerated (say, your etcd certs, or some weird one-off tool underpinning everything that somebody has to remember to regen every 360 days), they will expire, and it will be a very fun day at the office. Over a long enough period of time, your web server's TLS version will be obsoleted in new browser versions, and nobody will be able to load it.
It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done.
You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, especially after that first 6-12 months of "hardening" where bugs are found and patched.
Many people working in tech have never been exposed to a piece of software which isn't being constantly changed in small increments and forced upon end users. People are assuming that software is inherently unstable simply because they never use anything that isn't a "cloud service".
This probably comes off as "old man yells at cloud" but I'm not trying to bash cloud here. The cloud/SaaS approach has a ton of advantages for both consumers and businesses. But the average tech person in their 20s vastly underestimates how stable software can be when you aren't constantly pushing new features.
Absolutely. I remember we build a unimaginably brittle application many years ago, I think it was running on Windows XP and glued together a complex system with COM calls into this single page webapp even before react was a thing. It was build on a very small budget, serving the core business of a very tiny company.
Like maybe 8 years later I found out it was still humming along happily, without really even a sysadmin attending to it, on a single workstation using consumer hardware, servicing the company that had grown tenfold in size.
It blew my mind it still just worked all these years.
Your broad point is obviously correct (most outages are caused by code or config changes) but there are still classes of failures that can happen without any real changes, like various performance degradations (maybe your table grows too large) or occasional catastrophic failures from things like disk space or id overflow or something.
There's also the stability of third party systems: forced deprecations, security EOL, etc. The cert expiration stuff people have been mentioning is in this category too. I wouldn't be surprised if something does slip through the cracks at Twitter in the next 4 or 6mo.
I remember how it was over a decade ago and one hallmark of such systems was that they were easy to exploit.
My friend in college would just go into Wordpress admin panels and the like by using common exploits because nobody updated PHP on their VPSes back then.
As someone who spent most of their career to date as a front-end developer I learned that as long as they have the budget, stakeholders are insatiable. It's just that ten years ago most of their ideas were either technically not feasible or very expensive.
Nowadays browsers are much more capable, so the pressure to produce more features is much greater.
The other side of that is browsers. Even if you don’t change your code, the platform people are running your code in changes, automatically in many cases. New JS or CSS behavior in next safari or chrome? You need to patch/push to accommodate running environments that are outside your control.
The old space jam site worked for ages, and would still work if they hadn’t taken it down. The web is pretty good about keeping backwards compatibility.
The ecosystem changed since then.
Now updates of your binary's environment are more frequent, and often enforced. How often did you update or patch your Windows 98 or NT?
Today, it's false to assume that fire and forget releasing will work even for standalone Windows binaries.
Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.
We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough.
The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier and more foolproof than adding any sort of intelligent monitoring around it. I think an engineer added the cron job in the middle of the night after getting paged, and it stuck around forever... at least for the 6 years I was there, and it was still running when I left.
We couldn't fix the leak because the team that made it had been let go and we were understaffed, so nobody had the time to go and learn how it worked to fix it. It wasn't a critical enough piece of infrastructure to rewrite, but it was needed for a few features that we had.
USA at some point had an anti-missile system that needed periodical reboots because it was originally designed for short deployments, so the floating point variable for the clock would start to lose precision after a while.
I once managed a cluster of worker servers with an @reboot cronjob that scheduled another reboot after $(random /8..24/) hours. They took jobs from a rabbitmq queue and launched docker containers to run them, but had some kind of odd resource leak that would lead to the machines becoming unresponsive after a few days. The whole thing was cursed honestly but that random reboot script got us through for a few more years until it could be replaced with a more modern design.
This is a feature in many HTTPDs, WSGI/FastCGI apps, possibly even in K8s. After X requests/time, restart worker process. Old tricks are the best tricks ;)
You don't even need that, the kernel OOM killer would take care of this eventually. Unless its something like Java where the garbage collector would begin to burn CPU.
Unless the OOM-killer kills the wrong process.
Ages ago we had a userspace filesystem (gpfs) that was of course one of the oldest processes around and it consumed lots of RAM. When the oom killer started looking for a target, of course one of the mmfsd processes was selected and it resulted in instantaneous machine lockup (any access to that filesystem would be blocked forever in the system call which depended on the userspace daemon to return, alas never returning). Was funny to debug
Agreed, one of the craziest bugs I had to deal with was we had a distributed system using lots of infrastructure. Said distributed system started having trouble communicating with random nodes and sub-systems. I spent 3 hard days finding a Linux kernel bug where the ARP cache was not removing least recently accessed network addresses. Normally, this wouldn't be a big deal for a typical network because few networks would fill up the default arp cache size. That was even true for ours except that we would slowly add and remove infrastructure over the course of a couple months until eventually the ARP cache would fill and remove the random network devices... It wasn't even our distributed application code... Some bugs take time to manifest themselves in very creative ways.
Yeah, network scaling bugs are the most fun. The one I liked the most was when after expanding a pool of servers, they started to lose connectivity for a few minutes and then come back a minute or so later as if nothing happened.
Turns out we accidentally stretched one server VLAN too wide, to roughly 600 devices within one VLAN within one switch. The servers had more-or-less all-to-all traffic, and that was enough to generate so many ARP requests and replies that the switch's supervisor policer started dropping them at random, and after ten failed retries for one server the switch just gave up and dropped it from the ARP table.
Of course the control plane policer is global for the switch, so every device connected to the switch was susceptible, not just the ones in the overextended VLAN.
My team had a similar issue with the ARP cache on AWS when we used Amazon Linux as an OS for cluster nodes, and Debian for the database host. When new tasks were starting some had random timeouts when connecting to the database.
It turned out that the Debian database host had bad ARP entries (an IP address was pointing to a non-existing MAC Address) caused by frequent reuse of the same IP addresses.
Debian has a default ARP cache size that's larger than Amazon Linux (I think it's entirely disabled on AL?).
As for the tooling we used to track it down, it was tcpdump. We saw SYN's getting sent, but not ACK's back. Few more tcpdump flags (-e shows the hardware addresses) and we discovered mismatched MAC addresses.
We didn't have much tooling outside of typical things you would find in a Linux distro. It started with trying to isolate a node having issues. Then looking at at application and kernel logs. Then testing the connection to the other node via ping or telnetting a port I knew should be open. Found out I couldn't route then just process of elimination from that point till we managed our way to looking at a full ARP cache. Tested that we could increase the ARP cache size to fix the issue. Then figured out by going through the kernel why it wasn't releasing correctly by looking at the source code for the release we were using. I'm simplifying some discovery, but there was no magic unfortunately.
If resource leaks became a serious issue I imagine they could buy time by restarting. I'm curious what the causes were for code freezes. At Meta they would freeze around Thanksgiving and NYE because of unusually high traffic.
Same, code freezes were typically around holidays (when you know traffic will be elevated, engineers will be less available and you want increased stability)
I once debugged a kernel memory leak in an internal module that manifested after around 6 years of (physical) server uptime. There are surprises lurking very far down the road.
Back in the Pleistocene I worked in a ColdFusion shop (USG was all CF back then and we were contractors) and we had two guys whose job was to bounce stacks when performance fell under some defined level.
> you don't run out of disk space (from logs for example)
For a social media / user-generated content application, the macro storage concerns are a lot more important than the micro ones. By this I mean, care more about overall fleet-wide capacity for product DBs and media storage, instead of caring about a single server filling up its disk with logs.
With UGC applications, product data just grows and grows, forever, never shrinking. Even if the app becomes less popular over time, the data set will still keep growing -- just more slowly than before.
Even if your database infrastructure has fully automated sharding, with bare metal hosting you still need to keep doing capacity planning and acquiring new database hardware. If no one is doing this, it's game over, there's simply nowhere to store new tweets (or new photos, or whichever infra tier runs out of hardware first...)
Staffing problems in other eng areas can exacerbate this. For example, if automated bot detection becomes inadequate, bot posting volume goes way up and takes up an increasing amount of storage space.
> absolutely bring a site down over time, is expired certs
From today's Casey Newton's newsletter:
In early December, a number of Twitter’s security certificates are set to expire — particularly those that power various back-end functions of the site. (“Certs,” as they are usually called, serve to reassure users that the website they are visiting is authentic. Without proper certs, a modern web browser will refuse to establish the connection or warn users not to visit the site). Failure to renew these certs could make Twitter inaccessible for most users for some period of time.
We’re told by some members of Twitter’s engineering team that the people responsible for renewing these certs have largely resigned — raising concerns that Twitter’s site could go down without the people on hand to bring it back. Others have told us that the renewal process is largely automated, and such a failure is highly unlikely. But the issue keeps coming up in conversations we have with current and former employees.
I can imagine both cases being true, that the renewal process is automated and that certs won't get renewed because institutional knowledge has left the door. Where I'm at, service-to-service TLS certificates (the bulk of our certs) are automatically rotated by our deploy systems. But there are always the edge cases: the certificates manually created a long time ago (predating any standardized monitoring systems) with long expiry dates, and certificates for systems that simply can't run off the standard infrastructure. Sometimes, they'll bring down systems with low SLOs; other times, they'll block all internal development.
> the certificates manually created a long time ago (predating any standardized monitoring systems) with long expiry dates
Like the ever-popular "expires in 10 years" long-lived certificates. I've seen that happen: the VPN certificate, probably created by one of the founders 10 years ago when the company was tiny, expired one day without warning, breaking the VPN for all employees until it could be replaced (manually on every device).
They aren't talking about the front-end certificates which expire in Feb 2023.
It's likely the ones to encrypt all of the traffic involving the Finagle micro-services, data sources, observability systems etc. And I suspect the issue there is that you are going to need to do a rolling restart.
Which I personally would not want to be doing if 90% of the company is no longer there.
The way TLS was integrated into Finagle, most services should not need to be restarted to pick up and use their new certs. That said, there are certain core services that will require manual intervention, and there will inevitably be some services that should auto-update but do not.
> There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc.
In my experience as a data engineer, unusual states are one of the leading causes of issues, at least after something is built for the first time. You can spend half a year running into weird corner cases like "this thing we assumed had to always be a number apparently can arbitrarily get filled in with a string, now everything is broken."
Also, conditions changing causing code changes is the norm, not the exception, definitely in the beginning but also often later. Most services aren't written and done - they evolve as user needs evolve and the world evolves.
> As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine.
> ...
> There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site.
Aren't these changes inevitable, though? There is no such thing as bug free code.
Another thing that forces consistent code changes is compliance reasons- any time a 0-day is discovered or some library we're using comes out with a critical fix, we would have to go update things that hadn't been touched sometimes in years.
At my last job, I spent a significant amount of time just re-learning how to update and deploy services that somebody who left the company years ago wrote, usually with little-to-no documentation. And yes, things broke when we would deploy the service anew, but we were beholden to government agencies to make the changes or else lose our certifications to do business with them.
Eventually, Twitter will have to push code changes, if only to patch security vulnerabilities. Just waiting for another Heartbleed to come around...
Software never goes stale, it's the environment around it which stales.
Something from the 70s works perfectly fine, except it can't run on anything bare any longer, and the hard drives etc. have all long since failed or their PSU capacitors have blown....so Twitter will absolutely rot, how fast depends on several factors.
I personally suspect the infrastructure used to build Twitter will rot faster than Twitter itself, and of course the largest most dramatic source of rot is the power required to run it - several large communities have abandoned it already, making it less much less relevant, meaning the funding for it will also dry up, meaning more wasted cpu cycles and the like.
Thats of course assuming its left in some sort of limbo, it doesnt sound like thats the case with the current management, its only a matter of time before it topples over from shitty low-rate contractor code. Honestly, the app worked like so much hot garbage already, I could see it falling over itself and imploding with a couple poorly placed loops...
This assumes security doesn’t matter. You can’t run on stale code and be secure for too long, at least for anything non trivial. I imagine even if Twitter doesn’t add any functionality at so, it will still take hundreds of patches per Yasser.
Something from the 70's was not connected to the internet where millions of people are using it every day and finding every single edge case, or they are trying to break into it to steal valuable data. It was definitely not beholden to the same government regulations as a social media site running in the 21st century.
But real world conditions can force code changes. For example, a region abandons daylight savings time or a court order on copyright infringement. Someone unqualified working a system they are unfamiliar with could blow it up. Losing that knowledge of how the system works is a risk.
Wasn't last year where a bug in a Java well-used lib (and pretty easy to use) caused mayhem in every SRE/SEC services in every banks? It was a fun two weeks. I worked like 50 hours the first three days.
An example where something that correlates with time can reveal pre-existing bugs long after the system was chugging along just fine: counter limits/overflows.
Simple example: you have a DB with a table with an auto incrementing table. You chose a small integer type for the primary key and after years this just worked fine, you finally saturate that integer type you can no longer insert rows in the table. Imagine now this has cascading effects in other systems that depend on this database indirectly and you end up with an "outage"
> The biggest thing that brings down a site is changes
Absolutely agreed. In that vein, there is such a thing as too much automation. Sometimes, build chains are set up to always pull in the newest and the freshest -- and given the staggering number of dependencies software generally has, this might mean, small changes all the time. Even when your code does not change, it can eventually break.
It's been my experience that a notable part of software development (in the cloud age, anyway) is about keeping up with all the small incremental changes. It takes bodies to keep up with this churn, bodies which twitter now does not have.
It'll be interesting to keep observing this. So far it's been a testament to the teams that built it and set up the infra -- it keeps running, despite a monkey loose in a server room. It's very impressive.
How do your databases perform when their CPUs are near capacity? Or disks? Or I/O? I've seen Postgres do some "weird s%$#": where query times don't go exponential but they go hockey stick.
* Fan-out and fan-in
These can peg CPU, RAM, I/O. Peg any one of these and you're in trouble. Even run close to capacity for any one of these and you're liable to experience heisenbugs. Troublesome fan-out and fan-in can sometimes be a result of...
* Unintended consequences
The engineering decision made months or years ago may have been perfectly legitimate for the knowledge available at the time. However, we live in a volatile, uncertain, complex, and ambiguous (VUCA) world; conditions change. If your inputs deviate qualitatively or quantitatively significantly, you risk resource utilization issues or, possibly, good ol' fashioned application errors.
"No battle plan survives contact with the enemy." -- Stormin' Norman
Same with software systems. They're living entities that can only maintain homeostasis so long as their environment remains predictable within norms. Deviate far enough from that and boom.
I worked as an engineer for a very large non tech company (but used a lot of tech, both bought and in-house). We had 100s of teams supporting services, internal apps (web and mobile), external apps (web and mobile), and connections to vendors plus a huge infrastructure in the real world that interconnected to all of this. One time someone changed something in a single data center (I vaguely remember some kind of DNS or routing update) and every single system worldwide failed in a short time. Even after the issue was resolved, it took most of a day and hundreds of people to successfully restart everything, all while our actual business had to continue without pissing off all of our customers. The triage was brutal as to what mattered most.
You can't do this without a lot of people. Sure you could pare it down, maybe improve some architecture, but without a ton of people involved who understand the systems and how they connect, when things might go south they may never return.
I have an old project I gave up on - haven't touched it, done any code changes or maintenance in... almost a decade? At least a stubborn client is still using it, successfully. And it's not an old guy in a living room, but an honest small sized company that has this software as the core of its operations.
So yeah, I totally agree with you. No code changes = long life.
You didn't mention data scale. Just because the disks have room, doesn't mean the data access patterns in perfectly stable code will perform well at continual multiples if old data isn't somehow moved to colder storage.
> There are other things that can bring a site down, like [...] too much traffic[.] But generally speaking those things are rare and don't bring down an entire site.
I agree with your assessment, but I do want to highlight that this condition is not rare for Twitter. Load is very spiky, sometimes during predictable periods (e.g., the World Cup, New Year's Eve) and sometimes during unpredictable periods (e.g., Queen Elizabeth II's death, the January 6th US Capitol attack). It isn't going to cause a total site failure (anymore), but it can degrade user experience in subtle or not-so-subtle ways.
An aside on the "anymore", there was a time when the entire site did go down due to high-traffic events. A lot of the complication in the infrastructure was built to add resiliency and scalability to the backend services to allow Twitter to handle these events more gracefully. That resiliency is going to help keep the services up even if maintenance is understaffed and behind a learning curve.
Sorry for hijacking your expertise, but why no mention of memory leaks? In my experience they can cause really weird bugs not obvious at first, and are difficult to reproduce, i.e. triggered by edge cases that happen infrequently. Or are you assuming services automatically restart when memory is depleted?
It depends how well the service was "operationalized":
1) Best case: Monitoring of the service checks for service degradation outside of a sliding window. In this case, more than X percent of responses are not 2xx or 3xx. After a given time period (say, 30 minutes of this) the service can be restarted automatically. This allows you to auto-heal the service for any given "degradation" coming from that service itself. (This does not detect upstream degradation, of course, so everything upstream needs its own monitoring and autohealing, which is difficult to figure out, because it might be specific to this one service. The development/product team needs to put more thought into this in order to properly detect it, or use something like chaos engineering to see the problem and design a solution)
2) If you have a health check on the service (that actually queries the service, not just hits a static /healthcheck endpoint that always returns 200 OK), and a memory leak has caused the service to stop responding (but not die), the failed health check can trigger an automatic service restart.
3) The memory leak makes the process run out of memory and die, and the service is automatically restarted.
4) Ghetto engineering: Restart the service every few days or N requests. This extremely dumb method works very well, until you get so much traffic that it starts dying well before the restart, and you notice that your service just happens to go down on regular intervals for no reason.
5) The failed health check (if it exists) is not set up to trigger a restart, so when the service stops responding due to memory leak (but doesn't exit) the service just sits there broken.
6) Worst case: Nothing is configured to restart the service at all, so it just sits there broken.
If you do the best practice and put dynamic monitoring, a health check, and automatic restart in place, the service will self-heal in the face of memory leaks.
> If, for any reason at all, a cert fails to be regenerated (say, your etcd certs, or some weird one-off tool underpinning everything that somebody has to remember to regen every 360 days), they will expire, and it will be a very fun day at the office. Over a long enough period of time, your web server's TLS version will be obsoleted in new browser versions, and nobody will be able to load it.
At least for expired certs, most people have learned the hard way just how bad that is, and either implemented automated renewal (thank to heavens for cert-manager, LetsEncrypt, AWS ACM and friends) or where that doesn't work (MS AD...) monitoring.
I'll add one: when usage scales beyond anticipated levels. then that code that is "good enough" will no longer be, and serious intervention may be required - by senior engineers with history.
Takes me back to the first broken mess of an environment I worked in. Change freezes were a day of life and lasted and, magically, nothing would break during that time.
Now, those change freezes even extended to preventative maintenance, one of the dual PSUs in a core switch went bad and we couldn't get an exception to replace it... for 6 months. We got an exception when the second one went down and we had to move a few connections to its still alive mate.
I think that without code push they won’t be able to maintain compatibility. With updated APIs from third parties, new hardware, new encryption requirements from clients or browsers etc. It’s a slow descent into chaos indeed.
> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat.
I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually persuade me that it was all bloat.
> the article might actually persuade me that it was all bloat
First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject.
Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive.
If a product is mature enough, like Twitter seems to be, removing engineers won't instantly crash the product. It'll happen slowly. Bugs will creep in, because less time is spent on review and over all architecture. Security issues will creep in because of about the same issues and less oversight. Then, once this causes enough issues for the product to actually crash, the right people to fix it quickly might not be there anymore. That's when fixing the issues suddenly takes a lot more time.
If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. Especially with Elon pushing for new features to be implemented quickly, inevitably by people who cannot fully understand the implications of said features, because 80% of knowledge is missing.
By flowing from many people think it's bloat - I'll tell you what's really going on to tiny team of 1~3 built whole infra for critical component.
I'm not really trying to make commentary on whether or not Twitter engineering was bloat, or whether or not I think it'll hit problems in the future. Just commenting on the fact that the article broke my expectations a little bit as a reader.
https://twitter.com/atax1a/status/1594880931042824192 provides a bit more context and nuance (albeit with a sardonic tone, but it's hard to blame someone for being sardonic who saw their entire professional community fired).
There's no doubt that OP built a great and stable automation layer on top of Mesos for caching workloads. But there are numerous other types of workloads on top of Mesos (including, I presume mission-critical database deployments that need well-disciplined draining protocols to shift between nodes), as well as administrative needs for the Mesos-to-infrastructure level, and things running on bare metal below the Mesos level. These things all needed dedicated SREs, and the absence of these SREs could result in a scenario like the one mentioned in the Twitter thread I linked - two obscure mutually-dependent components expire and cannot be re-provisioned using documented tools.
I also think an important meta-point is that when Twitter was bringing in substantial revenue from advertising, every minute of downtime would have significant costs - costs that could make it easily worthwhile to "over-provision" SRE talent. With advertisers pausing engagement, perhaps Twitter loses less money from a day-long outage than it would save having the right talent to turn a day-long outage into a minutes-long outage.
Twitter is only judged by its profitability (namely, Musk's ability to service debt without selling more Tesla stock than he already has), while most other tech companies (both public and private) are judged by both profitability and revenue growth. If you want both, larger SRE teams, to say nothing of feature development and regulatory compliance teams, start to make a lot more sense.
> In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive.
It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk.
Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.
Sensitive workplaces (which seems like most these days) have taken to calling this the Lottery Factor (as in team members quit bc they hit the lottery) to spare more delicate types the pain of imagining their peers run over in traffic accidents.
Having been in the situation that a coworker did no longer arrive to work one morning because some person opened the car door when they were passing with their motorcycle, I can assure you that it's not only the more delicate type of people that all of a sudden are a bit more quiet than usual.
There's advantages to both terms. Bus factor maintains the (possible) fiction that people care about their coworkers/bosses enough that they'd transition out if they had the choice. Bus factor highlights that there's no way to get that information even if the person is nice enough to help for effectively free.
I work in the EEO/AAP space and can confirm our company calls it the Lottery Factor though it is partially because we do an office lotto pool when it gets big and jokes bled over. However it is much nicer for discussions. What will we do when employee x gets hit by a bus just does not sound good to random people walking by.
Almost lost a co-worker when he got hit by a bus five years ago. He's back at work full time now, with a substantial settlement from the transit agency (nice amount of f** you money). He wasn't an essential employee, but was important to our team in many ways. Fortunately he was always willing to joke about his accident. He'd survived, after all!
>Sounds like those people are in strong negotiation positions
You'd be surprised, but that's not necessarily the case.
One of my friends was such a person in a shoe making company as a designer. Instead of giving her a raise, they fired her.
Cue them re-hiring her a month or two later after they found out the hard way that the less experience subordinate really couldn't handle the job of the two of them on their own.
I know the bus factor is the morbid "how many people can get hit by a bus?" idea, but I actually like to present it as "how wide is the bus?" in terms of people being the conduits along which information and instructions flow; a wider bus provides redundancy. Aside from being more positive I think it's also more accurate for how we want teams to work. The "win the lottery and quit" metaphor is just stupid.
Have I been in a life-long bubble such that the ordinary explanation of "bus factor" seems so mild that until reading exchanges like this a couple times it never occurred to me it might be morbid enough to bother someone? Or is this one of those things where people are looking for something to worry about but it's actually entirely fine? Like, I've seen children's cartoons with jokes that were more morbid than that.
(I do also like the lottery version and use them basically interchangeably, though the shorthand is always "bus factor" for me)
I think it may be that some people have low environmental-sensation barriers.
For them, hearing "bus factor" may make it through the barrier and result in an involuntary creation of imagery in their head of someone getting gruesomely hit by a bus, and then the corresponding emotions they would feel (or simply just the emotions, without the imagery).
Phrasing "getting run over by the lottery" sounds like a good humorous, compromise. But "winning a bus" sounds like you're actively rooting for the person to be hit. I'd stick with the former. ))
I don't think the analogy holds in the same way. To me, the bus factor represents how many people you can lose for an extended period of time before there are no subject matter experts left for a particular topic. If you've got 8 people on the team, but only 2 of them know how to do a particular thing, the bus factor is 2.
It's not about having enough people to do the work even if someone quits, it's about having enough people that know how to do something that we aren't losing chunks of knowledge if someone quits (or dies, or gets fired, or gets sick, or etc).
It doesn't make sense to me to treat people as part of a conduit bus that are interchangeable as long as there are enough people.
>removing engineers won't instantly crash the product. It'll happen slowly
It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.
I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless.
With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerability comes out?
If Twitter went another month without an outage, how would you adjust your opinion? How about a year?
The car analogy is amusing, but how much does it really hold up? Have we ever seen another major social media company drop this much of its staff in one go? I certainly can’t think of an example. I think we’re in somewhat uncharted waters here.
A driverless car won’t last long, we know that for a fact. I think it remains to be seen how long a bloatless twitter can last. I’m personally optimistic.
> I think it remains to be seen how long a bloatless twitter can last.
It's also really hard to define 'last' though. Does 'last' mean just for up-time? Does it mean up-time without a major security incident while maintaining the same DAU? Does it mean business as usual on all fronts except number of employees? We know that Twitter already had some security issues with their God Mode admin panel.
I really wonder, for example, whether no angry ex-employees still have access to critical systems or data. This is usually pretty well regulated in large organizations like twitter, but since they've lost the majority of their staff, who knows when the people looking after that left?
> I think it remains to be seen how long a bloatless twitter can last.
I'm not convinced Twitter had a ton of bloat. (Most of the teams actually involved don't seem to think so). Just because Elon can't understand something, doesn't make that thing "bloat".
Twitter definitely had a few weird features that could be cut (the audio podcasting thing, for example). But calling most of Twitter microservices "bloat" is about as dumb as calling a cars Seatbelt and Airbag and Crumple Zones and that spare tire in the trunk "bloat" -- it's only "bloat" if you assume all people will always be perfect and no one will ever make a mistake anywhere, and nothing bad will ever happen.
I think it's well understood that when people refer to Twitter's "bloat" they're talking about its bloat of extra employees, not its extra microservices.
I've told this example before, but my close friend and roommate did a year at Twitter in 2019. He was tasked with implementing versions of simple, relaxing JS games like Tetris which would be used by Twitter's moderation staff when they felt they had accumulated too much stress and needed a break. He was making $300k USD to copy an open-source version of Tetris to be played by paid staff! This was actually one of his more engaging projects, most days he said that he wrote 0 code at all. He got great reviews and was being encouraged by his manager to pursue the senior track; from what I could observe from the outside, there was a complete misalignment of goals at pretty much all levels.
Perhaps we disagree which is totally fine, but I think this type of allocation of eng resources absolutely counts as "bloat". You may see that work as comparable to a seatbelt or crumple zone, but I personally see it as more comparable to all the other expensive, useless nonsense which plagues modern cars.
This is an interesting example but I have to wonder if it may inadvertently highlight a disconnect between the parties involved.
I'm no expert in this area but many stories have surfaced over the years regarding high levels of stress and (in extreme cases) meaningful declines in the mental health of social media content moderators.
I recently encountered a twitter thread from an early LiveJournal employee (1) @rahaeli "about Trust & Safety work, the toll it takes on you, the things you see, and the human misery, suffering, and death that happens when you fuck it up, including murder and child sex abuse." and I am inclined to believe that, while relatively simple and un-engaging, managers and end-users likely would have found it important.
> He was tasked with implementing versions of simple, relaxing JS games like Tetris which would be used by Twitter's moderation staff when they felt they had accumulated too much stress and needed a break.
Oh my god, this is incredible. Thank you for this story.
I have several college friends (developers) that ended up at twitter. Some of them loved it, some of them hated it, one universal thing I pretty much heard from all of them is that they typically did actual work 15-20 hours week.
Now overall I wouldn't say this is unusual in tech. I know plenty of people at other major tech companies, including other FAANGs, that say the exact same thing. But just because its appears to be a industry wide issue does not make it okay. It seems to me like that this would signify bloat. You only have enough work for your developers to be working half time.
Whats the problem with working only half of the time? Work is also a social enviroment and the «not working » part of work a realy important part. If the staff would work 100% of the worktime they would be fucked. Most people I know prefere a good working condition than working ultra hard for some sadistic asholes.
I don't even know where to begin with this comment? So you have no problem being paid for only half the time right? Or are you saying you deserve to steal from the company you work for?
There are a lot of assumptions baked into your position.
1. The employer pays for your time, not your expertise or output.
It'd only be stealing from the company if the company cares about hours worked over output. If we explore this concept in a theoretical sense it's clear that it doesn't hold up.
You have two candidates
One candidate has 20+ years of experience doing the exact thing you want them doing. This candidate says they'll work for you for $100k/yr, and they'll work 10 hours a week, complete all the relevant tasks, and very very rarely cause catastrophic errors or user-impacting bugs.
The other candidate is fresh out of school and says they'll work 50 hours a week. They'll complete the same amount of work as the first candidate, but they'll write more bugs, there will be more planning mistakes causing feature delays, and there's a reasonable chance of catastrophic failure due to debugging-in-prod shenanigans. They are also asking for $100k/yr.
Which candidate is better? Under the assumption that employers pay for _time_, the second candidate is better, but I'd argue most companies should prefer the first candidate.
2. More hours worked produces more or higher quality output.
There's a reasonable amount of research and practical anecdotes that disputes this recently (see companies that have gone to 32-hour 4-day workweeks with no reduction in productivity). Enough that at least, this point is seriously in doubt.
3. Twitter maybe pays a senior engineer $400k/yr under the expectation of their output for 40 hours, and if they get less it wasn't a fair deal.
This is a reasonable take, but Twitter (like most for-profit companies) theoretically has a performance evaluation system, managers, deadlines, etc. They're paying an engineer some amount of money for some amount of output. If that engineer produces that amount of output, Twitter is happy, the engineer is happy, there's no issue. If the engineer working 20 hours or less per week caused them to not meet their goals, then Twitter has the right to fire that employee. They don't, so that implies that they're happy with the arrangement.
4. The employee's salary is equal to their expected output/profit
If the Labor Theory of Value is correct, then companies derive their profits almost exclusively from the labor of their employees.
In order for an employee to "steal" from a company by under-producing work, they would have to earn more in salary + benefits than they earn the company from their work.
This is necessarily not the case (on average) in a for-profit company, because if the company makes a profit and uses those profits to grow or to issue dividends to shareholders, they have earned "_surplus value_" from the employees' labor (on average).
---
In any case, it's not necessarily true that you're wrong, but your comment was fairly dismissive and confrontational. There are a _lot_ of cultural and individual assumptions baked into how we exchange salary/wages for labor, and it's worth examining those before firing off moral judgments at one another for not working hard enough or working too much or whatever.
A lot of what you are saying is theoretical but doesn't hold up in the actual work place (at least in the US).
1. In the US at least we know a majority of the workers are hourly. (https://www.forbes.com/sites/johncaplan/2021/03/12/americas-...) So I would absolutely argue those people are paid for their time. I understand the scenario you laid out, but in a big corporation I do not think they would look at it like you are. They would simple look at who is costing the most. Both are costing 100k/yr so compare their output. Anytime their are massive layoffs, lots of important people are let go. This is because often times the people doing the firing, do not know the employees. They are simply lines on a spreadsheet to them. So in your situation above after a mass firing often times the candidate fresh out of school will be the one left.
2. I personally completely agree with you here. From my experience there is absolutely a burn out point. However major corporations do not see it this way at all. They absolutely believe throwing hours at problems brings about solutions. It doesn't matter to them if its making existing employees work 4 extra hours a day or hiring a new employee. It matters which is cheaper in the end. If the position is salaried its cheaper to have existing employees work more. Look at what musk is asking at twitter. If it is hourly, often times its easier to part ways with the burned out workers and hiring new ones. Look at amazons turn over rates.
3. I agree that Twitter gets to decide the expected compensation for work. But musk now owns twitter. He gets to decided what twitter does and doesn't expect from its employees. He made it clear he is very unhappy with how the arrangement was, and what he expects in the future.
4. I would disagree with you here. An employee can steal from a company many ways. For example, an employee could steal the source code from some twitter service that is considered a company secret. In this case, Twitter is paying an employee x amount for y work. Musk has decided what y work is. If an employee decides to not fulfill y work, and still take there whole pay, how is that not a form of stealing? If they're hourly employees they call it timesheet fraud. The idea that an employee can only steal if the are paid more than they bring in is pretty interesting but I think we would be hard pressed to find a single major corporation that views it that way. I think close to 100% of them would go after an employee spending 50% of their time working on non work related tasks.
A vast majority of the assumptions are based on them being the norm in corporate America which twitter is apart of.
> In the US at least we know a majority of the workers are hourly.
We're talking about IT employees. Unless someone was a contractor, most IT employees (white collar workers in general) are exempt, and not tracked or paid hourly.
I guess my question is - what are they doing for the rest of the week? Are they straight up just not doing anything, or just not writing code? I honestly don't think I could be writing code 40 hours a week, but I think that the code I do write when I'm writing it is the better for my taking the rest or zone out time. I also don't know if my work would be of a decent quality if I was suddenly asked to be typing all 40 hours a week.
Cannot speak to all of them but I know a couple work on sides personal projects the rest of their time. Doesn't matter if they are in the office or at home.
No one is asking them to write code 40 hours a week. That is not realistic in coding. Most developers' jobs include a lot more than coding. They are saying they spend 15-20 hour a week on all job related work. Whether that is coding, code review, engineering, testing, research, documentation, meetings ect.
maybe the thinking is tear it down to just bare bones and life support and then rebuild? Although, and I say this as a semi-fan, it's really exhausting trying to understand Musk heh.
Company soft killed the product, everybody left, they didn't hire anyone to replace. We went from 20 engs to 3. Worst codebase ever made by ex FANGs hotshots who thought they understood something about system architecture. Very "clever" and complicated. Data consistency issues happening everyday, likely due to misuse of messagging queues. Chargebacks being ignored and mailed in physical letters every month. A couple of millions going through the platform every year.
My task was to run a team of mostly juniors maintaining and adding features to that mess.
I had no clue what that codebase was doing. We just left things as they were, fixing fires as they came. Nothing too bad happened.
Slowly built a leaner replacement for some components. We simplified things over time and we even rebuilt some of the knowledge of the old platform, which helped with the daily outages.
The issues started happening not as often. Eventually. I moved on from that company, removing again a big chunk of knowledge. Over time I've heard tales of other people coming in and rebuilding that knowledge, over and over.
> Worst codebase ever made by ex FANGs hotshots who thought they understood something about system architecture. Very "clever" and complicated.
I call these sort of folks (very) smart juniors. Probably can whip leetcode or whiteboard tests like few others (after some preparation). Then you let them roam wild on your product, because they're of course experts. Then some period passes, and/or they leave, and you can only cry. Complex hard-to-grok approaches to simple problems that have tons of caveats/edge cases, no documentation of work done and why it was done as it was, because they are oh-so-cool and such lowly tasks are for peasants.
Either they have no clue how long term sustainable company looks like or they don't care, in any case not a good fit.
These days I go in opposite direction as probably most here - FAANG type of company (or cargo-culted startup) is a big fat warning sign when hiring. Unless I would be working at some wannabe another FAANG startup, which I am not, I would sure as hell make sure the person can actually deliver long term improvements for everybody and not just upp their CV with another bleeding edge technology and move, leaving more damage than added value and making everything worse.
> If Twitter went another month without an outage, how would you adjust your opinion? How about a year?
It's a tricky one, because on one hand it increase my trust that their system was built robustly, but at the same time the passage of time would increase the chance of unseen/unaddressed "wear an tear" (bot figurative and literal) that might be going unaddressed, or under-addressed. But we have no view into that.
We won't really know until they suffer a major problem whether or not they have enough staff yet to keep sufficient maintenance going that such an event doesn't cascade into something much worse and/or whether or not they will be able to recover from it in a reasonable amount of time.
Horrible systems can survive, but often they survive through sheer luck.
Beyond merely surviving, Twitter has to compete with other companies and enhance their product over time.
What happens if someone like Bytedance decides to launch a Twitter clone that does everything better?
Meta takes a lot of criticism for being Meta but you have to hand it to them, they've copied every other innovative social media feature that has shown up in competitors' products.
Does Twitter have enough staff to launch a major new feature?
The problem is that Twitter is now saddled with debt. It probably can't make investments in new products, and we can speculate whether it has the necessary headcount in the sales staff needed to maintain advertiser relationships that pay 90% of its revenue. [1, 2] Enterprise clients usually expect a dedicated CSM relationship.
That's exactly what happened to Toys R Us, which slid into liquidation due to the debt saddled on it by its private equity firm rather than a fundamental issue with its business.
In 2021, Twitter had a net loss of $221.4 million, and in 2020 they lost over $1 billion. Revenue was around $5 billion in 2021, about 90% from advertisers. [1]
The company (not Elon) is now loaded with $13 billion in debt. [2]
> Last year, Twitter’s interest expense was about $50 million. With the new debt taken on in the deal, that will now balloon to about $1 billion a year. Yet the company’s operations last year generated about $630 million in cash flow to meet its financial obligations.
>
> That means that Twitter is generating less money per year than what it owes its lenders. The company also does not appear to have a lot of extra cash on hand. While it had about $6 billion in cash before Mr. Musk’s buyout, a large portion of that probably went into the cost of closing the acquisition.
>
> That gives Mr. Musk little wiggle room, Mr. Pascarella said. “They are essentially going to take all the financial resources of the company and just pour it into servicing the debt,” he said.
I personally don't see how Twitter gets out of this without bankruptcy. I think Twitter can go the next 5 years without an infrastructure outage and still likely ends up bankrupt.
ByteDance I could see, but Meta and Google are hamstrung by the innovator’s dilemma to a degree: Both companies have the money and the technical smarts to pull it off, but their management structures would wreck it immediately because it poses a threat to other internal empires.
My loose guesstimate is that Elon has cut $2-3bn in expenses so far. I don't know the revenue hit yet, but $1bn in interest expense with $2.6bn in cash flows seems entirely sustainable
I think the income statement shows that it would be difficult for Twitter to find $2-3 billion of expenses to cut and still remain a competitive, functional company. You're talking about cutting total expenses by over 50%, including expenses that are not salary.
2021 total expenses of $4.8 billion on a revenue of $5 billion. $1.8 billion cost of revenue, $3 billion operating expense. Out of the operating expense, $1.175 billion is in Selling & Marketing Expense. $1.25 billion in R&D.
Let's say Twitter cuts R&D to $0, that's a $1.25 billion savings. How long can a social media company remain competitive putting $0 into R&D? Are there any examples of any software-adjacent company surviving that spends $0 on R&D?
If they cut SG&A, that will impact revenue negatively. Activities like marketing and sales have an ROI. You spend money to make money. Twitter was spending $600 million a year on SG&A in 2014 when they had less than half their current level of daily active users (DAUs). Twitter has a product, its daily active users. If they can't sell that product to its customer (advertisers) because it doesn't have enough sales staff to physically make the required phone calls (yes, they do that sort of thing with large advertisers), they risk entering the death spiral.
Cost of Revenue: I don't think this line item can be cut beyond a certain level. This is the direct cost of delivering the product. Anything that isn't employee salary can't be cut very easily. Twitter can't turn off servers and sell off data centers without impacting the product.
Twitter had more than 7,500 employees in 2021.
If they cut 80% of staff, that's 5,000 employees gone (and don't forget that those cut employees will still count as 1/4 of an employee for the next year due to the severance payment).
If each employee represents $300,000 in total expense (a very generous estimate), that's only $1.5 billion in savings, and for the first year the savings is only $825 million due to the severance payments.
Okay, maybe Twitter can raise revenue with Twitter Blue. They'll need to pick up a smidge over 15 million paid Verified users in order to cover the $1 billion interest expense. Twitter currently has 400,000 Verified users, ~240 million DAUs, so they need 6.25% of their global user base to purchase a subscription at $8/month. Out of Twitter's DAUs, you'll need to omit most users from countries that won't generally pay $8/month, like India, Brazil, and Indonesia. [3]
For reference, Spotify costs $1.46/month in India.
We could make a rough guess at this by assuming that the 155 million people in the US, Japan, and UK will pay $8/month for Twitter Blue, so you'd need close to 10% of the affluent user base paying for Twitter Blue.
For a benchmark, Discord makes almost all of its revenue from Nitro subscribers. In 2020 it had 14 million DAUs, with $130 million in annual revenue, which means with the $99/annual fee Discord had 1.3 million paid subscribers. In other words, about 10% of Discord users are paid subscribers. [1]
If Twitter can get the same subscriber rate with Blue, it'll reach $1 billion they need to pay off their loan interest, assuming Twitter Blue incurs no additional cost.
The problem is, it can't do that without that pesky R&D that we were talking about earlier. Discord is built around the concept that Nitro offers tangible benefits to paid membership. What features can Twitter build with a skeleton crew that will convince 10% of their users to pay for the service? Twitter Blue currently represents very basic functionality. [4]
I haven't even talked about the fact that I'm just talking interest payments for the debt, and the fact that Twitter wasn't profitable to begin with!
> Okay, maybe Twitter can raise revenue with Twitter Blue.
This made me laugh. Twitter Blue will have to be great for people to pay. Out of all your numbers, I wonder how many of those 155M users in the US actually post anything that would find value in Blue? My guess is that we're seeing Pareto in action where 80% of the Twitter content comes from 20% of the users.
I heard an interesting take today from someone in the social space. Basically as soon as they heard the price paid for Twitter, they knew it was over. It's just too much debt to both get out from under and move the company in the right direction.
I've started to come around that maybe it is that simple. Musk saw the debt payment and just cut the most easily cuttable expense in the short term - people. Bankruptcy seems inevitable.
I 100% agree with you. I don't even know how seriously Musk can try to avoid bankruptcy.
The leaked company communications are damning. Here's your brand new owner, richest guy in the world, implying the company might not survive another year or two, for a company that's been public, mature, and stable for years. It's completely insane.
Like I alluded to my comment, Discord's revenue story is essentially a best-case scenario type of benchmark. It's very unlikely that Twitter or anyone else in the social media space can replicate Discord's paid user share. Discord has been designing their product around a specific niche and optimizing for non-advertising revenue since inception. Twitter is designed for advertisers. It was never intended to be a paid product. They'll be lucky if 1% of their users bite.
I told you what I think he cut. I also said I don't know what that does to their revenues, but you don't either.
If you buy the narrative that 80% of employees were bloat and you can cut them without impacting top line, then it will work. If you think he's gutted essentially activities that will sink the ship, then it won't
> Twitter can't turn off servers and sell off data centers without impacting the product.
A friend of mine tried to tell me that Elon had some type of new computing technology that would let them turn off most of their servers because they could fit all of the tweets in less space.
He blocked me after I told him he had confused Elon Musk with gzip.
I think your friend gets at why people think Twitters IT costs can be cut though (ignoring his idea about compression): They think Twitters product is mostly the delivery of tweets.
But's not. Twitters product is ad inventory and their ad placement platform, and the engine driving engagement that boosts their ad inventory by reordering the timeline to keep people scrolling. Those are the hard parts.
"Just" delivering and storing tweets is easy. If you ignore the nasty business of the moderation. And most people have never even visited analytics.twitter.com and seen how much data is available to them about their own tweets, much less looked at ads.twitter.com and seen how precisely they can be targeted, and the precision Twitter offers in what kind of things you can pay for (engagement, follows, media views, clicks). And they've certainly not tried running ad campaigns in those categories, and seen how good Twitter are at showing your ads mostly to people doing what you're paying for.
Musk's actions makes me wonder how well he understood the complexity of Twitter too. Surely he must have looked at those other aspects of Twitter before he made his bid.
> If Twitter went another month without an outage, how would you adjust your opinion? How about a year?
I think it's hard to just use time as a measure. If there are no security issues and they add no features, then things should run fine. Which, ironically points to how solid the team was that was fired.
Of course if Twitter not only limps along, but thrives in this new setup then I'll definitely change my opinion. Being in the US, this might end up the case while Twitter for the rest of the world falls apart.
Keep in mind, that I think Twitter was bloated and needed a big shakeup. Randomly dumping people and those who tried to correct me is not the heuristic I would have used.
With that said, Twitter still has 2 huge problems. No vision and saddled with an enormous amount of debt. Right now, Musk is taking the PE approach to cut and milk what's there. The problem is, there isn't much to milk.
yes, maybe a car without mechanics rather than without a driver. drivers can be really bad, too. so bad that they forget to steer or steer badly, etc. but the steering wheel is categorically the interface made for non-engineer usage.
> I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless.
This is an excellently apt analogy, in light of Twitter's new owner.
That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.
Having been on the side of thinking I was absolutely irreplaceable on a team, I was more or less proven right when I left and the team failed to deliver anything at all for about 10 months, even having to shut down existing products, and is now limping along.
The truth in retrospect is that it was my fault (and my upper leadership's) that I wasn't replaceable. I created a knowledge silo around myself since I wanted to move fast and figured I could prevent the team from being bogged down in complexity if I just handled it myself and while that worked in regards to delivering out-sized results for the available bandwidth, it also was a risk that materialized as described above. So while I do believe that everyone should be replaceable and it's their responsibility to be, it's not always the case and products can live and die by it.
Saying everyone is replaceable is a first-degree approximation. The next level of detail would talk about, at least: (a) how much knowledge gets lost; (b) how many person hours -and- "wall clock time" it takes to rebuild that knowledge; (c) the amortized cost (or benefit) of losing that person
It depends on how well or poorly run the business is.
I worked at a company where everything hinged solely on one guy working from another country. When he left, loss of institutional knowledge took about three days to show real effects as things also came down crashing.
I worked hard to make _myself_ replaceable for when I left, it was a pretty good exercise, but me having that degree of freedom was symptomatic of the problems of the company.
how can you possibly say everyone is replaceable while next giving an example of a case where it's not only not true but potentially so invisible in its consequences as to make it hard or impossible to satisfactorily replace someone? a "person" can be rebuilt but not replaced. companies and societies aren't actually bose-einstein condensates. everyone is replaceable is just what different types of people say as a coping mechanism. it's a disgusting thing to promulgate too. maybe if someone is equivalent to a cog they can be replaced but humans are not exactly standardized cogs. what you mean to say, and may get in the habit of repeating instead, is that we don't necessarily need a specific person even though we are accustomed to them. it still might take you 10 years to find someone who does 80% of the same things as the other person while the missing 20% is what made that person such a unique hire in the first place. i find it a disgusting phrase that is so typical of the view of a society like ours today that is losing view of the value of life.
The dangerous piece of this is executives who don't have a solid grasp of how things are operating can assume that a product still running after the departures is still successful. Validates their decision making even as things deteriorate in the background.
That said you can replace people and build back that institutional knowledge -- both loss and gain take significant amount of time.
There are fairly widespread reports of technical issues. I’ve seen a few about 2FA failing, strange undefined behavior in the ad serving platform, and others.
Of course it could go either way but the jury is currently out. It’s entirely possible that severe company-impairing technical breakdowns are already in progress and unrecoverable.
I'm still on the fence. As an engineering manager, I tend to attach faces to those "jobs". So seeing cuts, I imagine a ton of people that had to go home and tell their friends/family/etc that they no longer had a job.
On the other hand. As an engineer, we tend to attach way too much self importance to our roles. Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is going to crumble. So... this is one I'm going to watch with a keen eye.
One need look no further than the O'reilly book on Microservices. First chapter I believe. We call ourselves "engineers" but the truth is, real engineers have real consequences when their designs fail. (buildings fall, bridges fall, etc). It's not a unique take. Now I'm not going to argue about engineering vs developing vs programming as that opens a massive can of worms.
>> Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is going to crumbl
> Never I have encountered an engineer that thought that.
there are people all up and down this page saying effectively just that. well i guess i'm assuming most of these people are engineers in the software sense of the word.
It's great to hear your concern with the actual people. That tends to get lost for some reason. And I agree we tend to attach too much importance to our roles, but the flip side bears a certain amount of truth, though on a timeline more like 108 days than 108 minutes.
I definitely walk a fine line of wanting people to know how much I value them and how valuable they are to the business as a whole. But, the harsh truth is, every single one of us is replaceable. I've just seen that play out too many times in my career. People with all kinds of domain tribal knowledge walk out the door. Everyone holds their breath as if it'll be the end. And a month or so later, we're still afloat. (I still hate to see good people leave)
Nowhere do I assert that people do not have the desire to accomplish things. Perhaps you're misreading my statement about "believing the building will fall down without us" incorrectly?
Yes. Your team members care about what they do, and think it's important. You don't think it's important and you view them as replaceable. Your intent is quite clear.
1. You're wrong. You have no idea the value I place on both my direct reports and other engineers at my current and previous companies.
2. I should have looked at your comment history before my first response. I really would like to assume good intent. I don't think I'm going to find it.
1. True, the only information I have about your perspective is what you've shared here.
2. I'm not sure to what you're referring?
When a manager comes on here and says "these workers think they're important but they're just shuffling numbers around" that deserves to be called out.
> The job cuts are clearly justified because of the extremely toxic "work culture"
I'm very curious in what way the stuff Elon is doing is not creating toxic work culture? >Minimum< 40 hour work week. 100% office work. Working into the night. Being fired for pushing back.
I'm not denying there may have been toxic work culture before, but I don't see how replacing it with another form of toxic work culture is to be celebrated.
Also, writing this:
> There is no active independent thought around this subject to be found. Only vitriol and bile.
And then spewing a bunch of vitriol and bile really distracts from any point you were trying to make:
> akin to witnessing a fat child throwing an epic tantrum laying flat on a mall floor
> Straight up Cartmenesque.
> clearly justified because of the extremely toxic "work culture" / cult
Up until recently 100% office work was the norm. It still is for most people. Many employers are quite adamant about it, not just Elon.
40 hours per week is also known as the standard work week. A bog standard j-o-b. Those are not reasonable examples of any sort of toxicity.
If these painfully regular employment terms are not competitive with other offers nobody is forced to pick twitter as their employer.
> but I don't see how replacing it with another form of toxic work culture is to be celebrated.
Walk on by. Find a company more suited to your temperament that will give you the pay and perks you think you deserve.
> And then spewing a bunch of vitriol and bile really distracts from any point you were trying to make
> because mommy didn't buy them an xbox
I assure you the child was quite literally stating xbox as the reason for his tantrum. Loudly. Whilst withering on the floor. I'm simply giving an eye witness account.
Toxic to me is joining a company under certain terms of employment only for some billionaire to come in and drastically change those working conditions for the worse. I frankly don't give a shit if others have it worse, there's always others that have it worse which is why I didn't take a job at somewhere worse.
You are basically describing every single takeover. When a company changes ownership it is unusual if there isn't a drastic shake up of things including working conditions. I don't understand why this one is a such a big deal other than the employees are throwing a temper tantrum. This type of thing is happening all the time all across the country yet people are discussing it like this is some new huge thing.
If as an employee you do not like the shake up, you always have the right to seek work other places. If you are indispensable (lottery/bus test) you may have some negotiating power but most of the time when up against a big corporation you are just SOL.
Every takeover? That's funny because when my employer was bought by a bigger company they increased benefits to ensure that they wouldn't lose their best employees.
yeah i think that's a perfectly reasonable view to have. If the job changes to something other than what you signed up for and you're unhappy then I feel like you have an obligation to yourself to find something else.
The 20-hour-full-time-from-home-six-figure tech job wasn't a new paradigm or a shift in c-suite work/life balance philosophy. It was 0% interest rates coupled with asset inflation leading to tech companies having more cash than they could possibly figure out how to spend. A transient financial anomaly in a whirlwind of larger economic conditions.
Now that the fed has been (rightfully)pulling the rug, people shouldn't be surprised if tech work goes back to the 60/70 hour grinds it was infamous for.
The projections were the work of Alan Marling, a Bay Area activist who declined to give his age or comment on his profession. He stood nearby — donning a Captain America face mask and a Sunrise Movement beanie — as passersby, many of whom were tech workers getting off work from nearby offices, stopped to take videos and photos.
> If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos.
It's not like Twitter was bug free before. How many times it annoyingly refreshed the timeline while I was reading something, or when it shows notification that it failed to send the DM, and when you retry it says "you've already wrote this", or you open the reply dialog, but it freezes, has no send button at all, so you have to re-open it. All of this was happening to me pretty regularly long before Elon came along.
As we all know, just hiring more people is not necessarily the solution to every problem, and to me it seems it was exactly what Twitter tried to do in the past. Now they deconstructed it to the bare bones, which will clearly show what are the core problems and requirements. They basically turned Twitter back into a startup. And from that new starting point they can hire again to cover the needs as they arise. If they succeed it will be a huge success as they'll end up with far more optimal team (and huge savings), and of course, if they fail to catch up with problems it will be a huge failure. We'll see how well Musk can manage it...
You're right on this potentially being the worse purchase of all time. The purchase price of 44 Billion was overvalued. If the only thing Twitter is supposed to do is to move tweets around then there was a ton of bloat of developers with a massively bloated purchase price. If Twitter had quality R&D products that could compete with TikTok, etc., then there may not have been a bloated workforce nor bloated purchase price.
Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?
I worked for a bank with 28000 total engineers. I know because part of what i did was crunching github and gitlab data and detecting double accounts (and deleting account from people not there anymore). I'd say 3/4th of them were just cost of doing business. Including me (my role wasn't really necessary, the stuff i did was cool and everything, but a bit useless).
At this scale, having someone take care of deleting double/expired accounts is just good hygiene and I would not consider that useless. Forgotten accounts are a security risk.
forgotten account that a former employee can re-activate or login with? I would think banks would be fined by regulators or risk their license over even a single instance of that.
Any off-boarding procedure fails - at that scale, if even one in one thousand fails, you can expect a couple of failures per year. ensuring these get caught is part of "solid offboarding".
That's not an excuse. If a bank can make sure that they do not mistakenly just deposit free money into people's accounts by mistake, they can ensure that a person who has left (possibly on bad terms) does not keep access to critical systems.
That's an entity that specializes in RISK MANAGEMENT.
There was a Twitter thread (ironically) on the front page recently where a Reddit founder explained why anyone will always suck at it. I recommend looking it up.
You need to pay for content moderates to prevent lawsuits. Lawsuits are very expensive but less so when you say we had thousands of people working on it.
It's just evidence that a thousand moderators (plus many, many engineers for automated moderation systems to reduce the load) is still nowhere near enough to not suck at moderation on Twitter scale.
the power law applies to any big organization. 20% of the people do 80% of the work, whilst 80% of the people are just there for "support".
whatsapp was run by a team of like 20 people or something when they got acquired for $20 billion. for a simple software product, you don't really need that many people. in fact, more people often means bad software. you just need a small group of very talented engineers to run the product and add new features when necessary.
big (and especially public) companies often times need to hire a lot, just to look like a real company.
now that twitter is private, elon has no responsibility to public investors and can focus less on looking like a real company and more on doing what needs to be done to cut bloat/costs and improve product
I've heard stories of teams that were more or less dedicated to nothing useful except policing the company, product, and users for woke violations. All these "affinity groups" to ensure voices were "being heard" when decisions were being made or some garbage like that. In essence, at some point there was a takeover and senior leadership either agreed with it or simply didn't want to get into trouble like Google did with these things, so they just went along with it all.
it showed up over night and now the phrase adult day care is everywhere. Where do y'all get your talking points from? Is it a rss feed I haven't heard about?
I'm a parent who has had children in day care for the last four years. The parallels between daycare for our toddlers and what many startups and big tech companies have provided and allowed to transpire are very clear to me and many of my peers. As a sibling comment stated, we've been using the phrase within our circles for many years. Those circles? Private group chats and friend groups. Mostly due to the climate in tech of mob mentality, groupthink, and the reaction against thoughts that run afoul of the orthodoxy of the week.
Your choice of wording suggests that you find the phrase or concept offensive, which would be understandable. But attacking the phrase as a talking point is shallow.
On the Internet, a new or rare term can suddenly start spreading like wildfire if it (a) is applicable to the topic everyone is suddenly talking about, and (b) perfectly captures something that people already recognise. It’s just life. It doesn’t mean everyone is an idiot or wrong or whatever. It’s just a word that works in the situation.
Yep, I understand how virality works. But virality works through networks. So when a word starts you can understand stuff about it through which network it originates from.
This can be something as silly as fan groups of a certain tv show, or as dangerous as propaganda from a foreign nation.
Adult day care has been used to describe tech companies for well over a decade in my circles. I usually saw it associated with start-ups in the past, but due to the recent trend of influencer-driven marketing videos (e.g. https://www.youtube.com/watch?v=AgaZsbXeddY but there are much more egregious examples) for large tech companies it has seen a resurgence.
what circles are those? I cannot seem to find any references to the term beyond this year.
> for large tech companies it has seen a resurgence.
it seems more like a bad attempt at insulting people who are trying to make work less shit. i will save my opinion at whether those attempts are more or less useful, but seems entirely another dumb culture war
I'm sorry, but allowing employees to put in a couple of barely-productive hours so THEIR work is less shit is utterly insulting to the people who strive to make companies function. Work is work -- if it wasn't, then it would be called something different. I would perhaps feel otherwise if compensation took this and a multitude of other intangible factors into account, but it doesn't so the derogative sticks for me at least.
I mean some jobs requiere more or less training. No one is a competent product manager at 22, so of course that girl is going through orientation and training and not doing 8 hours on her first day.
My senior PM, who has experience in several big companies, and was well aware of our team stll took a couple weeks to get up to speed with how everything worked etc.
I don't think her onboarding day counts to call Google an adult day care. And seeing things like free lunch and a treadmill desks are just tools companies like google use to keep you longer in the office and maximise your time not things to take care of you. Its more work, but mango flavoured.
Work is work, but the same way we think of weekends as normal, or 8 hours as a normal amount of time we could think about 4 days, or having a gym next to our work as work. Being sad at work isn't a feature in my opinion
https://twitter.com/anothercohen/status/1584636815281033216?... is a better example. It's not real, since this is a very poorly executed marketing exercise where they contracted an influencer-type, but it and videos like it are still driving the recent influx of "adult day care." Despite not being real, it's real-adjacent enough to not come across as satire. I have certainly witnessed plenty of people spend their days like this in tech companies.
Also note that I never personally said Google is adult day care, I was simply pointing to a culprit in the trend.
> still driving the recent influx of "adult day care."
perhaps we jsut have different definitions of adult day care. But that girl had a coffee and some snacks in the office. Which seems entirely reasonable. Being able to work from the rooftop might be a bit "fancier" but tbh I have seen dudes take work calls in rooftop bars for years and no one calls Morgan Stanley an adult day care.
I thought the adult day care was more related companies having talking sticks and mental health days. Which sure some do, and I can find it silly sometimes, but who knows how dysfunctional that team was and whether that works for them. I am sure if I explained to my parents what Agile is, they would also laugh in my face. Our work can look silly to others but work for us.
She didnt show any of her work but i dont think a video of her adjusting roadmaps and having a call with the stake holder of the API team their deadline is blocked by would really make for good instagram content.
I have a friend who was a mechanic, and he used to always do a drive around the block when a hard job (like an engine change) was done to make sure it was all alright. He would tell you about driving a nice car etc, but I didnt just imagine he didn't work the rest of the day, that was just the part he shared. Same reasoning with that PM. (Funny that both of them were PM thought).
idk the whole "adult day care" seems needlessly antagonistic for how people make work be less garbage. Specially holding against them things like free food which are "perks" only offered to keep people working longer
One problem with a humorous (yes, I am not all serious all the time) comment like this is while there could be some degree truth to it -- provide a source if so, please -- it smacks of exaggeration.
I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium.
That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm reading them, pushing the tweet I was in the middle of reading out of view. That seems like a pretty silly bug to exist in a mature product. I regularly reach a state where I have to kill the app and relaunch it because all of the "back" commands just minimize the app instead of taking me back to the timeline. I could go on.
They replaced their somewhat productive engineering workforce with completely unproductive interest payments. I’m not convinced that this will lead to a better spot.
But regarding the bugs, I’m totally with you. Same here. I use Twitter only in the browser. Browse long enough and the page reloads as if it ran out of memory.
Yeah, but there's always the option of showing a "fallback" when loading and inserting content in the middle of a feed - one link at the top "go to last bottom of top" and one at the bottom "go to last top of bottom".
Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?
It's bloat to devs who just want to write code. You're kinda confirming the point I was trying to make. Anyone on that list performs an important role at a company of their scale.
I'm currently the first and sole architect on a product that was built by only devs. I really know why I exist.
No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that.
Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…
There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.
Tens of thousands across all the organizations where postgres and clickhouse are used, maybe (though even then probably not!).
Not tens of thousands at any one organization supporting postgres and clickhouse.
No single organization needs even hundreds of people to support these apps. You just need a good architecture and a handful of dba's, developers, and sysadmins... maybe.. depending on your scale. At many smaller orgs you can probably get away with one.
This 100%.
You don't need 10'000 engineers to run billions of queries.
It's not like the SREs are running the queries themselves.
Your job as an SRE is to design and run computer systems, not to become the computing platform yourself.
I am entirely missing how Clickhouse solves the problem Twitter is solving out of the box, do you care to explain? Sub-question: how does Clickhouse provide monetization for a product?
You focused mostly on additive bloat, there's also multiplicative bloat in the form of multiple teams focused on building separate versions of the same product to increase likelihood of success and empire building where leaders don't actually have a remit large enough to support the team size they have, but they have woven a narrative that defends the necessity nonetheless. Put everything together and teams are very easily 6x+ larger than they absolutely need to be to get a product into market.
> I am convinced you can run the entire tech stack with a team of a 100 people
Please tell us in detail about the Twitter stack.
Because I always find it fascinating how people think they can estimate the effort to maintain it whilst having next to no understanding what so ever of the tech stack.
I assume the reason for statements like that goes something like:
1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple.
2. Modern devops tooling makes it fairly trivial to spin up 10k instances of a container instead of 1, by just altering a number in a k8s manifest somewhere.
3. Ergo, a single person equipped with modern tooling (and sufficient funding) could spin up any number of mastodon instances.
4. Twitter is just a big mastodon instance.
5. Now that keeping everything up is sorted, add another 99 devs for feature development and you are done.
Now this is obviously faulty logic because points 3 and 4 are very false, but they look reasonable enough at first glance.
That is a straw man. There are other perfectly valid interpretations and distributions of a statement that 100 could maintain it, like this:
15 database admins
10 linux sys admins
5 kubernetes specialists
10 windows tech support
25 front end developers
15 back end developers w/ Scala
10 machine learning experts
Whether that makeup could or couldn't do it is a different question, or whether it would be a different mix; all of that is up for debate, but the 1/99 ratio is just one very specific, extreme, and laughable mix for anyone who has supported a system of any real size.
How is that valid for creating and maintaining Twitter? 15 backend engineers are not even sufficient for any single facet of Twitter. Plus, there are other initiaves that Twitter takes on that are probably outside the scope of what you consider Twitter. Take a look at just their open source initiatives: https://opensource.twitter.dev/projects/. Go to, e.g., Finagle's Github page and look at the number of contributors and commits. While you may disagree that all of this is essential for creating and maintaining Twitter, there are problems that creep in when your platform supports billions of users that are not solved either correctly or at all by existing libraries and services; plus, open source projects serve as important recruiting collateral for engineers, especially for companies that do not have a set of widely used tools that entrench engineers (e.g. AWS, GC, Azure).
Creating a husk of an app that looks vaguely like Twitter and supports hundreds of users is a weekend project for anyone with a modicum of talent; building a platform that supports billions of users and is monetized well enough to support itself is an entirely different beast.
> 1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple.
There are some major pieces missing from your analysis: legacy code and infrastructure, and lack of good documentation.
These can make for a massive hairball of complexity that can swell the number of people needed to support it.
This reminds me of a talk I once saw by a Netflix SRE, who showed a crazy convoluted mess of a diagram with thousands of crisscrossing lines going everywhere, and him screaming "No one understands Netflix!!!"
If you think Netflix is bad, you should see architecture diagrams for one of the major ad platforms; 30+ service sub-systems get encapsulated as a tiny block of the massive diagram. Anyone who thinks complex systems can be built and maintained by a skeleton crew hasn't worked on complex systems or hasn't been exposed to their full scope.
also, iirc a database based architecture was why the failwhale image was made. iirc the threw out databases and went with something more analogous to email with tweets represented as files on file systems. That was was a lonnnng time ago though, they may have reinvented the architecture a dozen times since then.
I don't know why you are being down-voted. Premateralized flat files pushed out to heavy-duty CDNs would get you very, very far. I would think that dealing with CSAM, death-threats, and other things that would get your platform into trouble would be the more difficult problem to manage. I could very easily see the technical parts of Twitter being run with a few hundred people. I don't know what else would be involved for legal, finance, marketing, sales, etc. But I doubt it is 8,000+ people.
I couldn’t give you details, but I do know Instagram had 13 employees when Facebook bought them. Is Twitter really two-orders of magnitude more difficult to run?
Well, let’s see. IG was acquired by FB in April 2012 and had somewhere above 10M users at the time. Around the same time, Twitter had around 140M accounts in US alone, nearly 500M worldwide. Do we want to continue the apples vs oranges comparison further. Happy to keep citing scale differences. :)
Internal sources only, but topics like regionalization, localization, bespoke caching implementations, hardware-level optimizations, content moderation, policy adherence (e.g. GDPR, CCPA), long-term monetization (especially if supporting advertisers as a direct customer), 3P support, public APIs, documentation, SRE (a fledgling product doesn't need 5 9s), analytics (internal and for advertisers and 3P partners) and for the most part, security, are non-exhaustive examples of things you can mostly ignore when your product supports 10M users that are unavoidable when your product supports half of the US, a good chunk of the rest of the world, and other large businesses that consume you at large business scale.
> Well, let’s see. IG was acquired by FB in April 2012 and had somewhere above 10M users at the time. Around the same time, Twitter had around 140M accounts in US alone, nearly 500M worldwide.
So increasing your user base by one order of magnitude requires increasing the number of employees by more than two orders of magnitude? Rate of employee acquisition should probably never outpace rate of user acquisition, so I think that's a pretty clear sign that something was off.
Was instagram selling advertising to giant brands yet when they were bought? Sales staff balloon because they are really good at selling themselves to hiring staff and also because any brand that pulls in more than $10M in revenue expects to be treated like the only king of the world in pretty much every interaction and literally requires handlers, and the number of handlers required scales with the size of the brand.
You can potentially solve the handlers issue by being ruthlessly cutthroat. If they want hand-holding, they can hire a third party to manage advertising on Twitter. They probably already do in fact, so if you are a third-party, do your job and know your tools.
As for sales staff being good at selling themselves, agreed, so maybe Musk's ruthless firing spree will end up as a good thing. Maybe.
>You can potentially solve the handlers issue by being ruthlessly cutthroat
No, that just results in those businesses and brands leaving you, unless you can provide them a LARGE revenue stream that is impossible to get anywhere. A large brand will absolutely give up a little money just to spite you and your company for not treating them like god.
That's true, but also hardware and software hasn't stood still in the past decade.
I'd definitely like to hear more about the scale differences. So far at best, you've accounted for one order of magnitude. How do you explain the second?
In my mind the issue is less “is it conceivable that a small team can run a bare bones version of the Twitter app with hundreds of millions of users” and more “can a small team manage a big distributed system that was designed to be managed by many different teams, with no handoff period”.
The person I responded to was specifically talking about the tech stack. I wholeheartedly agree that the difficult part of running a service like Twitter lies in the soft problems.
But the profitability requieres a big tech team regarding ads and recomendation algos etc.
And moderation requieres humans but also a big tech team regarding bots, known offenders, etc.
it's not exclusively a tech problem but in a tech company tons of those responsabilities will be handled, addressed and solved by product and tech teams. And on 10 people, no matter how smart, they got no chance
Child safety advocates have praised Elon for his quick action in removing inappropiate child photos from Twitter, as well as dealing with the hashtags traffickers use. Something people have been asking for years for and little was being done.
Unless you wish to ask for people to openly post child porn guidance you're gonna have to rely on the people actually following the issue for years. For what it's worth business insider claims they checked said hashtags and the content was gone[0]. Said user is allegedly going to release today an article in a corporate outlet that did fact check it[1] so we'll see.
I have no reason to distrust her and can't see any reason for her to lie about it.
> Musk responded to the tweet saying that the issue is "Priority #1."
Obviously it's "top priority"! Is there any other acceptable answer? This answer means nothing except that he saw the question.
The reason for distrust is that the Q people have been insisting that Trump was secretly engaged in an enormous battle with pedophile rings, and using the typical, constant, and normal arrests of pedophile rings as proof. Now, another claim of a right-wing hero bravely picking up the sword and vanquishing the forces of evil, but again, no evidence! No proof at all. We just have to take her word for it, eh?
Of course, she can't give out the hashtags and allow independent verification, yet somehow hordes of pedophiles already know these hashtags? (How are they publishing those to each other, and if they have such a channel, why aren't they using that instead of Twitter?) So who is she keeping the hashtags from then?
>Obviously it's "top priority"! Is there any other acceptable answer?
the answer means nothing, yes, the supposed good action that triggered the question is the important thing. The only relevant part of the article is them claiming to have verified it, so it's Business Insider's word added to hers.
I'm not even going to touch the conspiracy theory madness, it's irrelevant, and the person in question has zero signs of being afflicted by it, her entire existence in the platform seem to have been focused on actually working against the issue.
Its also been pretty widely publicized twitter's issue with CP[0][1][2][3], and given how hastags are the way you find things in the platform it's natural that's the way they'd do it.
> Of course, she can't give out the hashtags and allow independent verification, yet somehow hordes of pedophiles already know these hashtags?
She is giving it out to journalists and apparently they are confirming it, it's not a tragedy that someone does not want to amplify possible child abuse, actually teaching even more pedophiles how to find it.
> How are they publishing those to each other, and if they have such a channel, why aren't they using that instead of Twitter?
Twitter is protected by its sheer scale, is huge making detection harder than dedicated sites, and is far far more stable, safe and accessible than some darknet website they'd have to run themselves on average.
> So who is she keeping the hashtags from then?
From the general public to not further humilliate the victims, from other pedophiles because its not some hivemind, the list goes on.
Not everything is about Trump, and you're falling for the same level of conspiracy if behind all this out of all this you see pizzagate Qanon.
I honestly don't understand how anyone reaches 2022, given all that we've experienced in the past several years, with enough faith in journalism remaining intact to pay any credence to "X happened, but we won't show you any evidence. Trust us."
For me, such a story is totally meaningless. It conveys no information about reality either way. The chances of truth or falsehood are exactly equal.
We have different epistemologies.
> from other pedophiles because its not some hivemind
Did you think critically about this? If so, for how long?
How are the pedophiles teaching the sooper-seekrit hashtags to each other? Is in the manual "So You've Decided to Become a Pedophile" that they send to new members of the vast conspiracy?
The problem with these conspiracy theories is that any inspection of how they might actually operate, day to day in the real world, is always neglected and handwaved away.
The whole point of a hashtag is that they spread virally, or are obvious terms. The very idea of a secret hashtag is an oxymoron.
When people say this it's because they assume there's not stupid ass politics or moat building.
THEY could probably do it with 100 people, YOU cannot.
100 people is most likely within the ballpark for a group of people whose sole purpose is to write and maintain twitter's tech stack. Unfortunately, that is not NEARLY the sole purpose of most people in businesses and that adds all kinds of productivity hits.
What happens is that people like yourself become convinced that's the only way to operate.
The question is not how many people to maintain an app with the functionality of Twitter, it's how many people to maintain Twitter. Twitter has to maintain Twitter's actual app, not an app that you built to be maintainable by 100 people. You can't say whether that's possible without knowing how Twitter actually works.
Not tech but I'm convinced that just answering LEO requests from all over the world takes at least 500 people. That's without their managers, payrolls, etc.
Likewise, bringing in Ad money would be a few more hundreds, because you need to chase leads in all countries.
Getting the Ads to work? That's tech and I'd be surprised if it was less than 100 people, too.
Tracking ads conversions, targeting, refinement of models, latency reduction , data analytics, ads sales, sales ops, etc. yeah easily 1000 employees all in.
I was only thinking of getting ads to work (from displaying ad previews to announcers to actually delivering them). If you add all the sales, analytics and performance, yeah, that climbs quickly.
For a second I was confused how Starlink requests were relevant for Twitter. Then I realized you meant 'law enforcement officer' not 'low-earth orbit'.
as someone who worked at an SSP, that is nowhere near what is needed for an established platform. There were a handful of engineers(10 if we are being generous including data science) keeping the lights on. The rest of the time was spent trying to prevent "senior" engineers from pushing code attempting to execute a sql query per each of the billions and billions of daily api calls.
Is there a reason to integrating ads into WhatsApp would require more than another 50 people? Twitter ads are certainly do not appear very complicated. The most complicated thing about Twitter is scale, which is why the comparison is made with WhatsApp.
> recommendations,
Does Twitter have recommendations? From what I understand, the front page was actively curated - that is, a human chose stories to put there. I guess you could count the god-awful default feed ordering as "recommendations", but there is nothing advanced about it.
> bots
If WhatsApp doesn't have bots, it's the only social media/chat app I've ever heard of that doesn't. What is needed for this other an an API?
> had to provide tooling for governments, regulators, content moderators etc.
I'm sure at least some of this exists for WhatsApp. Nevertheless, how many additional employees does this have take?
I am not sure why there is so pushback against the idea most companies are overstaffed. For the most part, yes, everyone has "work" to do. But most of the work is fundamentally unproductive. It's this way throughout the economy, but a few tech companies probably do represent extreme cases. I think the best argument for their case is that most of them are very profitable anyway (not Twitter, somehow), and they might as well throw money at thousands of people to do stuff in case one of them accidentally does something that ends up being wildly profitable. I am fairly neutral on the whole thing; I strongly dislike Elon, but I also think Twitter was horrifically mismanaged. While I doubt Twitter will come out better than it is, the idea that firing most of such a large organization would necessarily result in the immediate collapse of a mature product does not say much about the people that were fired.
I'm more sympathetic to the idea that it would get even worse over time, but I don't think there's anything necessary about this. You could focus on resolving longstanding issues while pausing most new work and probably come out perfectly fine.
> Twitter ads are certainly do not appear very complicated.
You see a couple of ads mixed in your feed; behind that there's a big machine selling that space to advertisers and mixing it into the timeline of every user based on whatever profile Twitter has created for you. Then the advertisers want to know how their ads are doing, or they'll stop buying them…and you'll probably need to have salespeople to get them to put money into your ad system in the first place.
> I guess you could count the god-awful default feed ordering as "recommendations", but there is nothing advanced about it.
Just because you don't like the ordering doesn't mean it's not advanced.
> I am not sure why there is so pushback against the idea most companies are overstaffed.
Twitter could be overstaffed. In fact it probably was overstaffed. But it's not overstaffed in the tune of of "it should be 10 people working out of a garage".
> You see a couple of ads mixed in your feed; behind that there's a big machine selling that space to advertisers and mixing it into the timeline of every user based on whatever profile Twitter has created for you. Then the advertisers want to know how their ads are doing, or they'll stop buying them…and you'll probably need to have salespeople to get them to put money into your ad system in the first place.
This is not crazily complex, bleeding-edge tech. This is something fairly well-understood and at any rate done by a lot of teams in a lot of places. (Twitter's ad profiling also seems awful. Maybe I am hard to pin down.) Probably the most complicated part is coming up with data to make advertisers think their campaign is working. (I am extremely skeptical most ad spend is actually worthwhile.)
> Twitter could be overstaffed. In fact it probably was overstaffed. But it's not overstaffed in the tune of of "it should be 10 people working out of a garage".
I agree 10 is too low for anything but bare-bones keep-the-lights-on-this-month maintenance, but it seems likely you could have a great and functional Twitter run by ~200 employees. I've seen more done with less.
Just as one data point that might tell you why you are misinformed - Twitter's AI team frequently publishes at the biggest venues in AI research and do a wealth of machine learning research on the data and processes they have. Some of that is used in advertising, among other things (recommendations, anti-spam, detecting abuse).
There are very few teams doing advertising at the scale of Twitter, saying "done by a lot of teams in a lot of places" is accurate just like "programming is done at a lot of places so why is programming hard".
No doubt you can have big teams doing highly complicated work.
That doesn’t mean your AI system performs better than a simpler one. Or that the system is useful in the first place (recommendations.) I’m not saying they were sitting around twiddling their thumbs. I’m saying the vast majority of Twitter staff were not actually improving the Twitter product noticeably to users. They were doing highly complex, cutting-edge engineering that was make-work.
If Twitter tech was so advanced, why were they losing so much money?
The complexity of your product has nothing to do with whether it is profit making or not. If that was the case, you wouldn't have loss making products in the AI space nor would you have profit making products in the garden shovel space.
Advertising is a hard problem that not many companies have solved at the scale of Twitter, that is what I am trying to get at. There are not too many social media networks out there which have hundreds of millions of users and billions of data points, and it's very misleading to say that work done in such a scenario is "something fairly well-understood and at any rate done by a lot of teams in a lot of places", when literally they're the only ones with Twitter type data outside of a couple of other Chinese social networks.
> The complexity of your product has nothing to do with whether it is profit making or not.
Yes, this is my point. All this incredible AI engineering did not actually make Twitter a better product. They could have just as well not spent the money. The work was ultimately futile for Twitter, even though it might have advanced our understanding of AI and have incredibly practical applications elsewhere. Conventional measures worked fine.
How is revenue the only metric for "scale"? It sounds like you really don't know what you're talking about if when comparing technical complexity, your metric to go to is how much money something makes and not how many user accounts need to be served or the geographical complexity of running a real time view consistent across the globe. By that metric, is Walmart or Saudi Aramco's tech stack more complicated and larger scale than a software company's?
This right here is why you can discount most replies on HN right off the bat. The "I can make software X in a day" posts are 99% bullshit because the posters making them have idea what business reality look like. If their program gained any popularity they'd be in a panic the first time the FBI dumped a warrant in their lap and their full stack developer is now spending a week with the lawyer trying to figure out how to untangle their data while the customers that paid for ads are yelling the metrics API went down 2 days ago.
Someone needs to interface with governments and law enforcements when they request data in criminal investigations. Someone needs to interface with lawmakers when new legislation is passed. Someone needs to handle data privacy requests from Europe. There's a lot of people working on this, or were at least.
Instagram had around 10M users at acquisition too when it was acquired a decade ago. IG has way way more staff now that they have scale. Must we continue to compare Apples and Oranges?
Agreed, there's decent starter comparables in the space.. IG, FB, DC, Goog all have public numbers. I've cobbled these together in the past week talking through with old friends from Goog and others. Please correct!
IG 13 employees at 30m users. Couldn't find # of servers.
FB had 10k servers in 2008 and 100m users, 850 employees.
I believe Doubleclick had ~500-1000 servers for ~10b daily impressions in mid-2000s.
Those numbers are all on circa 2010 hardware, so.. divide by a decade of performance doubling every 2 years (conservatively), or ~5x fewer servers in 2020.
The government takedown stuff, from personal experience, is tiny on the systems side; much more about moderators and expensive legal staffing.
These are very rough estimates, but I've heard 250k servers for Twitter.. that's much more on par with Goog/Amzn/Msft serving clouds at ~1m+ machines. That's a mystery to me.
What's their core business? Losing money on a platform where people can post racist comments?
Or do they have to earn money without getting sued for being used to spread CSAM and being a platform for harassment?
Because the first is very easy with 50 people. Elon can keep sinking money into it and never earn a dime (see Truth Social, they seem to be doing well!). The last is a lot more complicated and requires an ad platform, ad sales, content moderation, documentation writers, support agents, management, scrum masters, SREs, purchasing, et cetera.
IT people are really good at ignoring everything but the tech stack. Like tech is the only thing needed to run a profitable business. (It is... But not to run a 40B valued profitable business... And Twitter wasn't even profitable at all.)
> What's their core business? Losing money on a platform where people can post racist comments?
they've been running the same exact business for years now, if what you say it's true, maybe the answer is yes?
I never heard of Twitter making profit, so maybe their core business was "Losing money on a platform where people can post racist comments"
There was abundance of both, AFAIK, long before Musk
Remember when Dorsey tweeted nazi propaganda and then said his account was compromised? (which if true it means at Twitter they don't know what they are doing, if false, well...)
Remember when journalists wrote articles titled "Twitter is a Nazi haven for the same reason its CEO claims no bias" because Dorsey never actually distanced himself from the worst of the worst the platform hosted? Fearing he would be labeled as "too liberal"?
Remember when he started spamming crypto-bro propaganda?
Remember when a spy from Suadi crown worked at Twitter helping to uncover activists using the so called "free" platform and after he was discovered and reported to authorities, Saudi prince Alwaleed bin Talal bought 4.61% of Twitter shares?
Did the situation got better with more and more employees or it just revolved around banning prominent accounts? (not that I necessarily disagree with the reasons behind it, but if that is the best solution they've found, after years of fine tuning, they could have done it before with much less people involved)
> Or do they have to earn money without getting sued for being used to spread CSAM and being a platform for harassment?
it's easily provable that the 7,500 employees did not improve things on that front.
> IT people are really good at ignoring everything but the tech stack
that's a really odd proposition.
Looks to me that Twitter was in bad shape already, despite thousands of non tech employees, Facebook it's in no good shape either, despite tens of thousands of them, basically the only thing still working as intended in those companies is the tech stack.
I guess the real question you're asking is "why non IT people are so bad at doing their job"?
Not my opinion though, I never said IT over other departments, I simply said nobody is able to explain what makes WhatsApp so special that a hundred people can run it while Twitter requires 75 times that and still doesn't work as well.
Chats have a finite upper limit of participants, some accounts on Twitter have 100m+ followers. Storage is limited to buffer of not yet delivered messages, and avatars/stories.
They are entirely different technological challenges.
Sure, but nobody is upset if he sees twit 1 min later than someone else, since most of the time you can't even tell, a lot of people would bitch about 1 min latency on chats.
So twitter can afford to deliver those tweets with higher maximum latency than WhatsApp.
And it's scaling when you need to keep low latencies, that really kills you, at least in my experience.
WhatsApp and twitter’s latency calculations are on different things.
Twitter’s latency stems from calculating what tweets should show on a given request. Even if you try to show tweets from 1 minute ago, it’s hard to cache that stuff using traditional systems because of the fan out. If an account with 50 million followers tweets, you need to update 50 million timelines. How do you do that quickly?
And you would have to define maximum latency, is it seconds, minutes, hours? because you can’t have the timelines be inconsistent for too long as that leads to some people getting news faster than others.
now you have to deliver them, exactly one time, to each recipient or groups of recipients, through different network topologies, with different challenges and vastly different bandwidth and latency guarantees, in exact order, while also keeping track of who is online e who's not, and distributing that information in real time, only to the edge nodes that should know about it, all of that fully E2E encrypted but stored (indefinitely?) in case the recipient is currently offline and unless that recipient blocked the sender.
let's agree that both companies solve hard problems and that it's not the technical difficulties that make the two companies sizes so different.
right, because WhatsApp is a small company that makes very little money and has virtually no users and it mostly does not work nor scale...
We could all write that in a single weekend, if only we had no family to spend time with.
It's interesting how the prospective shifts when people are told "yeah, that's impressive per se, not very impressive compared to what the others are doing"
It's like discounting Sputnik 1 because the Russians did not employ an army of people selling ads, but just the people necessary to launch a satellite in orbit, which is actually the real achievement.
Anyway, from the news: Nearly 1200 software engineers left Twitter last week
Suddenly the Twitter engineering team sounds not so capable, which clearly is not the truth, the truth is that if you have hundreds of managers, you'll end up with hundreds of small teams competing to boost the ego of the manager, usually wasting thousands of man hours on miniscule returns (if not losses) while those power point slides will help someone else to get promoted for the new project that nobody uses.
Been there, done that, I don't know why a demographic so well versed in the dichotomies of the tech industry such us the users of HN is so baffled by the claim that 2,000 engineers for a single company that does what Twitter does is a complete waste of human potential.
Elon Musk is a person I would never work for and I think he's not even a good entrepreneur, but one thing he does right: he calls the shots and then executes them.
He said he would fire people and he did, many helped him by leaving on their own, which left Musk with the responsibility of proving he was right.
If Twitter will still be up and running in a year time, we can be sure that there were 1,200 engineer too many working there.
because, honestly, who really believes that the "influencers" will actually leave for the fediverse, where they'll have to work hard and compete with mere mortals, while they could keep cashing from advertisers to promote shit to their already established audience?
nobody believes that.
Also because the fact that Twitter will sell less ads in the next future doesn't mean that advertisers won't spend that money on Twitter, they will simply not pay Twitter, but the Twitter users. For them it's exactly the same thing, for Twitter celebrities it's a giant opportunity.
1 message broadcasting to 10 silo'd people is a completely different system than ensuring a celebrity tweet gets to 100 million people within a couple minutes, which is what twitter considered their task.
Who is defining and managing the PRDs and the roadmap in this scenario? What about user testing and customer development? By “graphic design”, do you mean User Interface design? If so, who is responsible for the user experience beyond just UI?
The point I’m trying to make is that it takes some effort (beyond just the plumbing) to create an experience that folks actually want to use on an ongoing basis
> Who is defining and managing the PRDs and the roadmap in this scenario?
Mainly the project manager.
> What about user testing and customer development?
Testing could be done by everyone, mainly the project manager, although you could perhaps add a tester to the team. Don't know what customer development is.
> By “graphic design”, do you mean User Interface design? If so, who is responsible for the user experience beyond just UI?
Yes, UI design. The graphic designer would be primarily responsible for UX, but also the project manager and developers.
This is based on my experience with mobile app development. Right now, it's a fairly complex product, users often say that it's much better for many use cases than the alternative by Google, which I guess may be developed by a 10x larger team.
We are not talking future development, just maintaining status quo.
Rode map is just: keep it working, or whatever EM twitted about hour ago :)
But I agree it's way to optimistic number. You need at least a few for each platform, just because of bus factor, you also need people to keep in touch with Apple/Google reps etc., making sure bills are paid etc.
I imagine it will mostly be just minor tweaks and no major features, you can easily do both mobile targets by cca 10 people, not working crazy hours. There are plenty of successful apps with smaller teams, that make it work.
Eh, tossing the direct comparison to Twitter out here because they seem to have gone too far the other way, this is much like saying "I can make 1,000,000 screws a day, so assembling at least 100 cars a day should be easy".
Well graphics design isn't hard. I'm sure one of the Devs could do that. And I'm sure there are tools to convert an iOS app to/from Android, so one dev can do both.
And a website is easy. You could do it with 1 person.
But Elon is such a machine, he could keep it running by himself.
You still have to read them to work out that user X saying they can't log in is the same as user Y saying authentication is failing at A stage, issue is present on B and C platforms version D.E
Further the issue of 'site using the wrong font' is not equally as important as 'site is exposing private user data'. And you're likely to get many people reporting the former, and maybe only one reporting the latter, so a sample isn't likely to catch the really important stuff.
Indeed. Just translating it could be a team of at least N+0.5 people, where N is the number of languages you want to translate your product to (the +0.5 accounting for translation verifications). Even if you outsource it that just slows down feature deployment IME.
The people shouting loudly about how Twitter must have been so bloated are really just shouting their obvious inexperience working at global scales or their localized ambitions.
Could there be too many employees at Twitter? Sure. Most companies have dead weight.
The number who were "extra" is probably not 9/10ths the employees though.
> I am convinced you can run the entire tech stack with a team of a 100 people.
This is because you don't see the complexity. What you see as a Twitter user is a fraction of what's actually there.
You have to build a platform for ads. Not just serving ads, but allowing advertisers to prepare their collateral, preview them, get their results, and be billed. So that's an entire content and invoicing platform separate from your main feed.
And since your platform is all user generated content, you've got to build a moderation pipeline. A place for users to make reports, but also an interface for your content moderators to view content and make decisions. Oh, and while you're there you'd better build a portal for law enforcement to make data requests, along with your DMCA takedowns. Oh yeah, DMCA - that's another whole thing you've got to worry about.
Then the EU comes along and needs you to build something to support your GDPR obligations. Then India wants something similar, but only for its citizens. Your users also want verification, so better build that platform for securely verifying accounts and awarding checkmarks.
It snowballs. Was Twitter's engineering group bloated? Probably. Most large companies are. Could you run the whole Twitter tech stack as it exists today with a hundred people? Absolutely not.
Ads isn't just serving, but targeting. The better you target, the more your ads are worth. When you have $5B of advertising, a 0.01% improvement is breakeven for $500k of fully-loaded comp. So you should add as many engineers / data scientists / etc as can generate an 0.01% annual improvement. Or maybe you want to take 3x their annual salary: that's still an 0.03% improvement in ad relevance.
Separately, some commenters here are flatly delusional about the effort to ship a site, android and ios apps, internal mod tools, help docs, support, and legal docs in 34 supported languages. Not to mention obeying laws in all the countries that implies.
Or image and video hosting! With recoding of videos, resizing of images, and the management of what is surely petabytes of images and videos with very high reliability! That is not a 1, 2, or 3 person job to do well.
A lot of commentators are either not engineers, incredibly poor engineers or utterly ignorant to how much work it is to manage a global company, as you've mentioned. Doing anything at the scale of Twitter is a nightmare crossing multiple languages, laws, domains and expertise much of which we'll only see start burning when governments step in and start taking chunks out of Twitter.
Anyone who has used Twitter, have you seen any evidence they do this beyond extremely basic geographical targeting.
Like people keep listing off all this stuff when we’ve all used the site and can see if it does have a team working on it then they’re not doing it to the levels of their competitors.
>> "These are some of the interests matched to you based on your profile, activity, and the Topics you follow. These are used to personalize your experience across Twitter, including the ads you see. You can adjust your interests if something doesn’t look right. Any changes you make may take a little while to go into effect."
They're not good at identifying interests and making them targetable, but they try.
To be honest, growth. I work at a tech company of a similar size, and keeping the lights on, so to speak, from a systems perspective is something we can do with a few engineers. By far the bulk of employees at my company are in R&D, sales, business operations, infrastructure growth, etc. If you build a resilient architecture, keeping something like Twitter simply running wouldn’t take that many people.
>If you build a resilient architecture, keeping something like Twitter simply running wouldn’t take that many people.
It takes an army of engineers to build a resilient architecture at Twitter's scale.
And why are we even talking about "keeping the lights on"? Elon is claiming he's going to build a better video platform than YouTube, complete with better tools and for creators, for crying out loud.
He also said we’d have robo taxi’s by now. I’m not sure how much faith you can put into anything he says because so little of anything he has said has been true. Hell, if he told me I was fired, I probably wouldn’t believe him. Sadly, HR probably would.
i see your point but you should always put > 0 faith in Musk IMO. No robo taxi's but, even after being laughed out of the room quite literally, landing orbital class boosters is common place now. I don't doubt there's the talent at Twitter required to do amazing things and Musk seems to find ways to push and motivate talent to crazy levels of results. I'm not saying things like a more youtube than youtube is destiny but don't completely write it off.
I didn't say 0 faith, I just said I wasn't sure how much. :p
SpaceX is an interesting result, but it is starting to look more like a fluke than what we'd expect from him. Perhaps rocket scientists "get him" and understand the stakes better, after all, if you screw up a rocket you could die in the aftermath.
Tesla is certainly similar stakes -- except one mistake can kill every passenger on the road instead of your fellow employees. Twitter is basically 0-stakes, and is unlikely to take itself that seriously. If they screw up, maybe a bad government won't get overthrown somewhere, or trolls will take over the internet. But mostly, people will survive -- or rather, the outcomes are far more abstract.
He doesn't seem like a good fit for this kind of environment.
A script that takes text from input field and displays it along with previous submissions in chronological order is trivial; the difficulty comes from serving millions of people.
For US only? This sounds like the comment complaining about app size when an Uber engineer showed up and talked about handling 100s of methods of worldwide payments. Many in the US seem to think the internet only exist in the US, and that US rules apply everywhere.
Googling "how many people work at stack overflow" gives a number less than 400. If true I think you could put together and run Twitter with less than 400 people from an engineering stand point. However Twitter probably has a much higher soft skills head count for moderation and what not. Though it wouldn't surprise me if a lot of that was contractors not direct head count.
I don't see any comparison at all. Does Stack have to maintain mobile apps? Their moderation is mostly handled by users, but Twitter must write and maintain tools for handling that. I doubt anyone is trying to upload illegal porn to Stack. Latency of 30 minutes would be noticed by very few Stack users. Twitter probably really needs at least 10x the SREs of Stack.
Yeah, but that 400 is everyone at stack not just engineering. A team to put together a mobile app should be less than 10.
>Their moderation
Above I assumed their moderation team was probably larger than their engineering team, and mostly contractors. Thus I kept my estimate to the size of their engineering team.
The whole point of the article is twitter was designed to be resilient. (and it shows, twitter has great uptime). And the whole point of resiliency, beyond not negatively impacting customer experience, is to buy engineers time to fix things when stuff breaks.
What we are watching is a massive failure event right now and the question really is if there's enough time for twitter management to fill in the gaps before there's an outage.
if 80% of knowledge is missing due to 80% of people being gone then the team who built it failed to document or automate themselves out of a job, meaning even other people on the team in the good ol' days would still have faced the same issues and merely had to chase down the original authors. that doesn't sound like it would be completely true. maybe 5-20% of un-documented knowledge walked out the door? completely rough guess. but even 0.5% of knowledge can sometimes be critical.
There's something poetic about NASA sending a rocket to the moon at the same time as Elon Musk is fucking around with Twitter instead of sending a SpaceX rocket there too.
I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by the parent, not trying to make one myself.)
Once you get the automation going the number itself doesn't matter that much.
You might have 200 different apps (hell, we have close to that, only 3 people in ops) but competent team will make sure they deploy in same way and are monitored in same way.
And once you go from "a server" to multiple servers, whether the end number ends up being 20 or 200 isn't that important till you start hitting say switching capacity, and if you're in cloud that's usually not your concern anyway.
Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself"
> Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself"
Nobody is adding changes that blows out the DB? or add some inefficient code that burns CPU much faster?
It's not 1-3 people. The entire SRE team globally - including the technicians and the engineers with server access - is easily going to be in the hundreds.
The SRE manager is in charge of keeping it all running. He isn't running around the world swapping out servers. He also isn't sitting back with his feet up thinking "All done - now how are my Pokemon doing?"
It's a dynamic process with quality monitoring, budgeting and reports, post-mortems, continual experiments to see if uptime can be improved, and redesigns as hardware and software change.
It's part of the backend, but is only loosely coupled to the content management and delivery system, the ad machine, moderation, marketing, and so on, all of which are going to have similarly complex structures.
it doesn't follow. The article posits that "many people think twitter headcount was bloated" then proceeds to describe a (presumably) really efficient work of a small SRE team. These two parts seem completely disconnected from each other - neither one proves, disproves or follows from the other - so it's unclear why the former was mentioned at all.
His personal experience was zero bloat. He was one deep in a critical function for the company. He isn't saying this proves without a doubt that there is no bloat, but he didn't see any in his time there. It's seems like a reasonable addition to the conversation to me.
It's a shame that most the conversation going on here are extrapolated arguments based on this article and another anecdotes. The problem starts when the ones making their points let beliefs on "how things should be" stronger instead of "how they really were".
No, I think the article makes it very clear what the value and function of SRE is. The point of the comment you're responding to is that the author was the only one doing this—not a team of ten, not even a team of two. This is Twitter's whole cache system! Probably the most important part of their hardware stack, in terms of "is the site performing well for users". There are other SRE needs at Twitter, but not that that many. What were the other 9k people at the company? It begs the question.
software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly.
"software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly."
In an organization of any appreciable size, things change all the time.. and I'm not just talking about code (for which you could have a code freeze in an emergency situation like this), but the external systems you're connected to could change for reasons completely out of your control. Content changes can break stuff because of bugs in your code. Legacy systems could require all sorts of ongoing tweaking and maintenance. And, yes, heat can break your software if the server it's running on overheats.
Agreed.. but lets say you fire 99% of your engineers, and declare a code freeze (because there's no-one left to write code)..
Then in theory.. if you own the hardware and you've locked down the libraries... That code could keep running for a long time. Agreed it's not a Palm app, but with everything locked down, I'd argue it's safe
But now I can third party stuff changing. Payment processors and such. Those don't happen fast though, and 100% not so fast that a company the size of twitter can't work out a sunsetting.
To the heat can break software if the server it's running on overheats. I have a feeling twitter's has a system in place to scale out the faulty server.
My point was, comparing code to a car is silly. A car needs maintenance. Code in code freeze does not.
Software bit-rots. That app from 1998 doesn't interact with today's world. As the world evolves around us, needs change and software has to change to keep up. That's not to say there aren't companies out there that rely on some ancient Windows 98 software program running on similarly ancient hardware, because there are. But Twitter as a piece of software isn't some static thing. It's needs are constantly changing and the software has got to keep up.
Your PalmOS app doesn't run on any modern hardware except under emulation. (Which is sad, I loved my Centro and held onto it for as long as I could.) The last release of PalmOS was in 2007, 15 years ago. Most hardware from that long ago is dead, and thus your software is dead. broken down by entropy to the hardware.
Agreed. But my comment was in comparison to a car needing maintenance. If nothing changes and I drive my car for 5 years without taking a look under the hood, it will be a mess. If not a stitch of work is done on it, I'm in trouble.
If however I have an app and I don't look under the hood for 5 years, it could still run as good as it did when I locked it down. As you said some companies run on apps written for windows98. Those apps are still working as they always did.
I don't think it needs are constantly changing. Like it could freeze for weeks/months. Leave existing bugs and put versions in lock.
I do agree that it will eventually need to change, but that's where selective hiring comes in. Oh system X isn't great. Lets find a team for that, all else remains black-boxed.
Even discounting external changes, any reasonably complex system needs maintenance because time moves on and new interactions happen.
How many SSL certificates (internal or external) need re-issuing per month? Some of that can be automated, but in an organization as large as complex as Twitter some will be bespoke and manual, and a code freeze won't stop the clock.
How many new CVEs per month apply to Twitter's services and tooling? How many race conditions or other bugs are lurking, just waiting for the right time or traffic pattern to emerge? Twitter can't freeze inbound traffic without dire consequences.
Twitter is like your car, except that it's always running.
To be honest I was very surprised to hear what a cache SRE was working on. It sounded like he had to build all of handling of hardware issues, rack awareness and other basic datacenter stuff himself. Does it mean that every specialized team also had to do it? Why would cache engineer need to know about hardware failures at all, its datacenter team's responsibility to detect and predict issues and shutdown servers gracefully if possible. It should be completely abstracted from cache SRE, like cloud abstracts you from it. Yet he and is team spends years on automation around this stuff using Mesos stack that they probably regret adopting by now.
I feel like in this zoomed in case of twitter caches what they were working on is questionable, but the team size seems to be adequate to the task, so my takeaway is that like any older, larger company Twitter accumulated fair amount of tech debt and there is no one to take large scale initiative to eliminate it.
>As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.
On the other hand, a car without a second and third steering wheel, 20 windscreen wipers, and an oven in the back, keeps running just fine, even after the first bend...
Have you also taken into consideration the death rate in smaller planes, much higher than commercial craft. The moment you attempt that with a 'high capacity' craft you'll suddenly find yourself regulated out of existence by a book written in blood.
I'm not saying they're bloat. They're a tradeoff between cost (man hours of work) and lives. They've increased the cost of small planes tenfold but have saved tens of thousands of lives.
And it's almost inevitable. That's why the overwhelming "Elon is so dumb" narrative seems so odd to me.
"Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users. Even if Elon literally has to re-hire half the roles he fired for, or Twitter is down for a few days or a week, or they struggle to get advertisers for a little while, that's nothing for the long-term. In six months, the chances that it'll look like these firings were a bad idea are minimal.
And likewise with the "the only realistic way to moderate an online platform is the way Twitter was doing it" narrative. In six months, all it takes is the ship to still be floating without the old moderation to prove it out, and that's by far the most likely outcome.
> "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users.
With the takeover, Twitter has been saddled with massive new debt and would have to become significantly more profitable to service it. Instead, they are losing ad revenue, and any savings from reduced head count will not manifest four at least 4 months because of the severance payments for the laid off workers.
Musk could of course keep the company going as a hobby for a few more years, but he strikes me as more likely to cut his losses early — preferably by finding somebody to take the problem off his hand, but, if necessary, by bankruptcy.
> Even if Elon literally has to re-hire half the roles he fired for
I have this nagging suspicion that the kind of people who line up for an opportunity to work without equity, lousy job security, and send weekly code printouts to the CEO are not necessarily the highest quality hires in the field.
> or Twitter is down for a few days or a week
The Twitter user base has quite a bit of inertia keeping them from moving to competing platforms — lots of people I follow have made backup plans on Mastodon, but not many have moved their focus there. But Twitter becoming literally impossible to use would probably be the one thing precipitating such a process.
> they struggle to get advertisers for a little while
And after that? Are the major brands going to decide that they like advertising on a Gab-lite Twitter after all? Or is Twitter going to strike it rich with ads for dietary supplements, gold coins, non-perishable meals, and tactical pants?
> In six months, the chances that it'll look like these firings were a bad idea are minimal.
The targeted firings were bad enough (To paraphrase someone else "Imagine ranking engineers by # of lines of code added, and then keeping the ones with the HIGHEST number"), but the voluntary departures are likely to be even more consequential. After the way the workforce was treated, what are the odds that the people left are the "best", rather than the ones with a lack of alternatives (e.g. for visa reasons), or Musk fans who may or may not be qualified?
This is exactly why this thread is full of slightly insecure comments making vague predictions. I'd suggest to most of them to get off HN and back to work, now is the time to make yourself useful!
Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.
No, it really isn't. It is perfectly normalizable.
Every twitterer is a newsletter. Most hardly ever tweet and sporadically at that.
The followers are subscribers. They hardly even see most tweets they subscribe to as the whole thing is quite ephemeral. Same as never reading all your emails (very few people are inbox zero freaks).
The timelines are just that, an email inbox. It is very soft "real-time" at best.
Tweets older than 48 hours can just be archived to a blob store and served as a static website. Most people consume it as such, logged out and from the browser.
You think brainyquote.com is hard to scale? Think of twitter.com as unbrainyquote.com.
All the other froufrou built on top of it is more complicated but 2007 style twitter is dead simple. That is its (completely accidental) genius.
There is no shortage of systems that are x10-100 times more "web scale".
The approach you're describing (fully materialized inbox model) simply doesn't scale. I say this as someone who worked on social media databases and scalability for a decade. Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system.
At minimum you need to do a hybrid approach which special-cases the more widely-followed users. This problem has been well-known for quite a long time. Yahoo's "Feeding Frenzy" whitepaper came out in 2010, but the concepts were definitely known before that; I remember hearing about hybrid activity feed designs in 2009 from colleagues who formerly worked on LiveJournal.
> Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system.
I knew somebody would bring that up. Currently there are fewer than 500k "verified users". Not that many people have 1 million+ followers and they don't tweet all that often.
> At minimum you need to do a hybrid approach which special-cases the more widely-followed users.
Great, so we are in agreement that for 99.75% of users it is all quite trivial.
I'll tell you what has happened since 2010: hardware has gotten a lot faster. 1 million iops is not a big deal anymore. Keeping that in mind you should refresh your assumptions.
We built a 2U Intel Xeon server system capable of 80 MILLION 512B random read I/O operations by combining the latest 3rd Generation Xeon Scalable Processor (code-named Ice Lake) with Intel Optane SSDs.
122M IOPS in 2U, with > 80% of the system idle. Easy.
Just to put this into perspective, at 4K random reads, this is 144GB/sec of bandwidth from storage, at 36M IOPS.
Fancy 512b random reads? You now get ~120M IOPS.
You could saturate streamed network traffic on 11-12 100Gbit NICs.
You can't just hand-wave away those edge cases. That's not how anything works.
If your potential write amplification for a single operation is anywhere remotely near a factor of 1 million, you have a serious problem and need to completely change your approach to the problem, and use different data structures and algorithms.
Hardware hasn't gotten that much faster really. PCIe flash cards started to get used over a decade ago -- yes, modern storage is better than those were, but not by a huge multiple. And meanwhile max CPU frequencies today aren't much higher at all. What we have instead is more cores. And a lot more RAM per box. Faster networking, sure. But none of this lets you get away with massive write amplification from choosing an overly-naive algorithm.
And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency.
I'm not making "assumptions" that I need to "refresh". I've literally spent the majority of my career working on this stuff at extreme scale, both in 2010 and today, and all times in between.
> And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about.
The twitter firehose is usually bellow 50MB/s. 200GB a day of tweets will bankrupt no one. An 100TB Nimbus Exadrive that does 100,000+ iops costs about $30,000. 1 year of tweets. Thousands of twitter employees fired probably saves $3+ billion/year in salaries, I'm sure they have a hefty enough hardware budget.
> Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency.
Does it really? I don't think so. Not with tweets.
For DR you can stream into a blob store like s3 in the background and have an automated process that stands up a fresh shadow cluster from it every couple of hours. Hardly costs anything with this volume of data. That is cold data, doesn't need the fancy blazing fast storage.
That's a single copy of all tweets. Not fanning out to up to 100 million inboxes. Completely different problems.
You keep citing raw hardware speeds of a single machine, yet we're talking about the feasibility of a distributed system being able to sustain random bursts of write amplification factor of 100 million across a decentralized database, with ideally exactly-once write semantics even if a failure occurs mid-way -- and that's all in addition to whatever the normal baseline write activity of all "normal" users with more reasonable follower counts. Again, completely different problems.
> I don't think so. Not with tweets.
So in your design, if the singular data center that maintains all users' inboxes goes offline for a long period of time, the entire product just goes down. And you think that's acceptable for a business valued in the tens of billions of dollars?
You seem absolutely convinced that a massive social network can be run on a shoestring budget with tiny staff, and no amount of evidence from someone like me (who actually worked on this stuff in depth, and posts with my real name, and expertise in profile) will convince you otherwise, so I suppose I should just stop replying to you.
> yet we're talking about the feasibility of a distributed system being able to sustain random bursts of write amplification factor of 100 million across a decentralized database, with ideally exactly-once write semantics even if a failure occurs mid-way
I think my assumptions are just a lot more relaxed than yours. This isn't a trading platform I don't see why you need exactly-once write semantics.
There are only 6 users with 100m+ followers and they avg a lot less than a daily tweet. @BBCWorld is #50 and it drops to 38m accounts. #1000 has 2 million followers.
> And you think that's acceptable for a business valued in the tens of billions of dollars?
Elon by his own admission grossly overpaid for it. Twitter has hardly ever eeked out a profit, it is not worth tens of billions of dollars. Nothing much would happen if it went down for a bit except maybe bored journalists would report on it thus, as ever, driving even more users to the website. But that is neither here nor there.
More to the point: if Elon and Obama and Bibster tweeted in the same minute (what are the odds) you would, gasp, have to stagger the fan out of the updates. That's alright too, for Twitter. It isn't really actually real time.
Those follower counts are also grossly inflated and as you understand yourself only a small fraction of them are online using the app at the same time as the person is tweeting. By the time they do check they might never even see the tweet.
To the people offline you don't need to fan out in a timely manner.
In short I believe the write amplification is much closer to 1 million than 100 million even with the pathological cases. And beefy enough hardware can handle those peaks.
Here's another way to think about it: Elon has 118m followers and just posted twitter has 260m daily average users. He is a bit like Tom from MySpace, half the users on the website are subscribed to his updates (not exactly really but for simplicity).
I think it is perfectly alright if it takes a full minute until all those users see his latest meme. It is very unlikely that even a quarter of all his followers are using the app during that exact minute, so we're talking 30m writes in 60 seconds. Big whoop.
> You seem absolutely convinced that a massive social network can be run on a shoestring budget with tiny staff
I would bet a budget of say <$1 billion/year and 100 engineers for the core functionality as is.
> no amount of evidence from someone like me (who actually worked on this stuff in depth, and posts with my real name, and expertise in profile) will convince you otherwise
Neither one of us presented any evidence, just opinions as outsiders, as part of an informal conversation. An appeal to authority isn't an impressive argument, I am also from this industry and with similar experience.
There is no need to take things personally. I think we just have a very different estimation of just how much activity twitter sees at peak and how strict the requirements are.
> I don't see why you need exactly-once write semantics.
World leaders use Twitter. It's a major international one-to-many communication platform. If tweets are lost or duplicated, it makes the platform look unreliable (because it literally would be) and as well as potentially making the tweeter look incompetent for posting twice. World leaders don't like to look incompetent, that can cause really bad things to happen...
> @BBCWorld is #50 and it drops to 38m accounts. #1000 has 2 million followers.
Even a write amplification factor of 100,000 is extremely problematic for the fully-materialized inbox model. A lot of prominent twitter users have followings larger than that.
> To the people offline you don't need to fan out in a timely manner.
So now you're adding additional systems on top, in order to scale. That's good, I guess you're starting to see that the problem is more complex than just spraying out every tweet to every follower's inbox. Now consider that when you actually build and scale a system like this, you'll need to keep doing that in a bunch of different areas, and the complexity keeps snowballing.
> And beefy enough hardware can handle those peaks.
There's no way to fit every users' fully-materialized inbox feed on one machine, so we're definitely talking about a large distributed storage tier / database here. Will you use "beefy" hardware for every single shard of your inbox storage tier?
> It is very unlikely that even a quarter of all his followers are using the app during that exact minute, so we're talking 30m writes in 60 seconds. Big whoop.
Once again, this really isn't like doing 30m write ops on a single box. It's queueing the writes via RPCs across a huge storage tier, while also needing some way to handle timeouts, retries, failovers on either side of the operation. All while the "normal" background level of thousands of tweets per second is happening from everyone else.
> An appeal to authority isn't an impressive argument, I am also from this industry and with similar experience.
> There is no need to take things personally.
I've literally built a reverse-chronological social network activity feed implementation, which successfully scaled to over 110 million posts/day. (For sake of comparison, Twitter was around 500 million tweets/day at that time, so this was def smaller than Twitter, but still quite large.) It did not use an inbox model. Took many months of my life, some of the most rigorous work I've ever done. My teammates and I evaluated several alternative designs, including fully-materialized inbox, running all the numbers in depth and building several prototypes. The takeaway was that a naive fully-materialized inbox would be completely and ludicrously infeasible in terms of necessary hardware footprint.
Separately, I've also spent years working on database infrastructure at extreme scale, including one of the largest relational database footprints on earth. I have a very good sense of what this requires. Yes, I'm posting "opinions", but they are based on many years of direct personal expertise.
Scaling a social network involves a massive number of challenging problems. Faster hardware doesn't magically make these problems go away. And while I haven't worked at Twitter, up until this month I knew four infra/backend engineers working there, and they're some of the best engineers I've ever known in my 17 year career.
I'm taking your comments personally because your comments are offensive. You're blindly saying I need to "refresh [my] assumptions" about a topic I'm literally an expert in. You're claiming Twitter could use some completely asinine overly-simplistic feed model, as if no one else ever thought of that, which would strongly imply every infra engineer at Twitter must be an idiot. In another subthread on this page, you wrote "The job cuts are clearly justified because of the extremely toxic work culture / cult" and it is necessary to "replace every single person who worked there and the entire tech stack". Seriously, WTF? These are hard-working humans with lives and families, they don't deserve this shit from their employer, and certainly not from offensive pseudonymous randos who have no idea what they're talking about. Have some empathy.
A lot of things in a typical social media service can be segmented trivially. The hardest part is the feed because it requires querying many database servers at once, but then it can be cached and served quicker.
Isn’t cache invalidation the hardest part of the feed? Once a user with millions of followers tweets, doesn’t that essentially invalidate millions of caches? That doesn’t sound like a trivial problem.
Because you can't edit tweets, the database is monotonically increasing (append-only) so scaling it isn't that hard. Cache invalidation basically just adds a new item. Deleting a tweet actually would be the harder/more expensive operation, but it's also less common.
Editing a tweet is also easy because the feeds would only store its id. You'd only need to invalidate one cache where the tweet object itself is stored.
That small team seems to have been running the caches for other teams, by using infrastructure provided by another team, in two massive datacenters operated by other teams, using monitoring tools managed by another team, and a ticketing system run by another, on hardware purchases by another team…
All just to put caching in front of services that actually do anything.
That being said it's not like twitter is a massively complex product with lots of different features. I can imagine you could keep it running with a skeletton team. Liasing with ads buyers excepted.
a bunch of people I talk to say it is massively complex but typically fail to explain how, especially given the super-glacial pace at which they added new features for 15 years. And yes, this article kind of doesn't disprove the bloat at all, unless every single SRE quit? but that is also not what's stated in there.
I'm pretty sure one of the most complicated things is preventing automation of content - bots. which would be an arms race type condition. Bots you want to prevent - people not using twitter's api to do their bot stuff. Why would people not use twitter's api?
1. who would trust twitter not to change API and make code worthless
2. people who want to do stuff Twitter doesn't want you to do in an automated fashion.
Even "simple" things become complex when they need to scale to billions of users around the world, handle a high rate of traffic, deal professionally and legally with all the different jurisdictions they operate in and all the advertisers they serve, etc.
I imagine the infrastructure implemented is fairly complex, this post does outline that it's not a super simple operation. It also alludes to there being more business units than just the core application.
> I was partly expecting the rest of the article to explain to my why exactly it wasn't just bloat
Same here. I guess his header was on point in why Twitter is still up; but I was also interested in hearing about why Twitter actually needs all those people. If it can be run with 50-80% of the staff gone, that does sound like some bloat at least.
Slack space leads to innovations, like developing infrastructure automation and improving capacity planning. SRE as a practice needs slack space for operations teams to work on improvements and fixes in addition to BAU fault fixing, deployments and patching.
I'm certainly no supporter of "lean operations" with minimum staff etc, and fully agree that you need people that are well rested (for the lack of a better analogy) to do great stuff. But I do think that some of these internet giants do have to many people working there; wasn't LinkedIn 14_000 strong when Microsoft bought it?
I've always felt that the American model of doing business is based on how we optimize network traffic, i.e. double the amount of data until failure; then turn in down a bit. Fire on all cylinders until people are truly worn out, then turn down the pace a bit. Haven't worked in the US so I'm pulling this info out of thin air...
Well, half of the staff prepares for some conference, meetup, tech talk, or helps organize one, or does 20% time, or sits in unnecessary meetings, or sits in necessary but inefficient meetings, or is on PTO or on unpaid leave in some retreat.
Plus above a certain headcount the communication overhead becomes seriously large, so just to compensate for the lost velocity you need to break out into smaller more agile more autonomous teams, which further increases the coordination requirements (thus the comms overhead), but allows overall throughput to scale.
And the leading edge technologies commonly used require a large headcount to begin with. (I mean just to start running something twitter/linkedin sized requires at least 1 engineer/million users, so a few hundred folks is a given. You need someone who understands networking, from BGP to TLS to VPN to whatever, internal IT, CI/CD, ops/SRE ... at that scale if you use anything, you need an expert for it. You use Kafka with a hundred millions of users? You might need at least a few people who actually know what the fuck a partition means. Unless you want to just directly give all of your money to Jeff in the form of egress fees, you might need folks to setup CDNs, and whatnot.
So without naming names, ina big password manager company (around a hundred million users?) a few years ago there was a certain rewrite project. 3+ people worked on it for 8-10 months, then it was put on hold temporarily. And then obviously nobody speak of it ever.
It happens that there are inefficiencies that for months not one line goes into production from certain individuals.
It was bad management, yes. But if good management was easy to find then we would be talking about different things :)
I'm puzzled by this statement. Do you think of resiliency as waste? That twitter would have been fine without it?
The article makes a point that the reason Twitter is running ok on 20% of personnel at this moment is exactly because it was build to be resilient, not because the personnel was bloated. A large part of this so called bloat, the 80%, was responsible for Twitter to be running right now. Calling this bloat implies it is actually not important for Twitter to be available all the time (or at all).
>>If anything, the article might actually persuade me that it was all bloat.
Not for me
This is almost exactly like the new manager coming in, noticing that the floors and surfaces are all clean, all the systems work, the trash is emptied, etc., and so deciding that the entire maintenance staff is unnecessary and firing them.
The place doesn't become a decrepit pigsty the next morning; it slowly degrades.
Same for these systems. They were designed, built, tuned, and maintained over the course of years to go from requiring constant manual intervention to running largely unattended and with a good buffer of ready hardware and automatic failover for failures. That "largely" in "largely unattended" is doing some very heavy lifting.
The system WILL require human intervention to keep running, and more than just a skeleton crew. The only question is whether it will happen before the new crew gets up to speed to handle the inevitable degradation.
This does NOT mean that the SREs were bloat - it means that they were doing an excellent job and could safely take a break. We're now in just the two-week vacation zone - same as if the entire SRE team went on a holiday. We'd expect it to work. Now let's see what happens in two months.
In addition to the many great comments here, remember that super star engineers don't exactly fix problems from day to day. They fix the problems before they become problems.
The engineer was doing stability planning for 6 months out for the purpose of cost optimization. I guess we can assume that the costs of infrastructure is about to go up and reliability is about to go down in the coming months.
I have been under the good faith assumption that most (though definitely not all) of the employees that have departed Twitter were probably necessary and valuable to the company. I left the article with the same impression as you. This single person did this very important job, seemingly well, and didn't appear to be drowning in the work. What were the other 8-9k doing?
> But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures.
... for the Cache component. There are many others.
Musk might have been right about some things. There probably was some degree of bloat. But to say he's badly mishandled this whole saga is a gross understatement. It is very difficult to utterly kill a site like Twitter; the fact that we're even considering that as a realistic possibility shows just how badly.
I think Musk is used to Tesla and SpaceX, which are both companies that a lot of people are (or at least were) excited to work for because they believe in the mission and what's being created. Plus there aren't many alternatives if you want to do that work. Twitter really isn't like that for most people; a Twitter developer has many other options to do similar work. Add to that the fact that he's both cranked up the intensity of the abuse and that it's more visible to everyone, and you can't expect a lot of good people to stick around. And despite the fact that it might coast for quite a while on the back of excellent work in the past, eventually you do need good people to keep a business going. (This is leaving aside the direct impacts of his actions on users and advertisers!)
>the fact that we're even considering that as a realistic possibility shows just how badly.
Do we have much to go with to "really consider it", or is it just sensenationalist headlines, as Musk went ahead the accepted orthodoxy on Twitter and content moderation and so "he must pay"?
That would be the same news outlets that built him up as the real life Tony Stark and propped up his quite ordinary companies, one of which is building electric cars like everybody else, just doing it in style as well-off upper-middle class people's toys, and another is doing space tech that was already a thing we were pursuing 40 years ago with a little modern engineering thrown in...
>Why aren’t the other space programs/companies landing rockets?
Economics, pure and simple.
SpaceX was heavily capitalised during an era of easy money, so they have cash to burn advancing this technology. Everyone benefits. But there is no indication they are anywhere near profitability. So why would other space programs do this, when they can reap the benefits (tech and cargo) of a VC and government subsidised company?
Or do you think the SpaceX engineering team is the only one on the planet capable of doing this? I'm sure they're awesome, but that's a stretch.
Perhaps they haven't gotten as good NASA contracts and subsidies as Space X.
Note also how "launching rockets" or even going to the moon and back, was something we routinely did 40 to 50 years ago.
Heck, we even tried reusable rockets, and even had space shuttles. They were cancelled for political reasons, just like moon missions and NASA programs were. Not for lack of progress or technology.
The shuttle being “reusable” is ridiculous when compare with falcon 9 or heavy. The entire booster pack was discarded and the shuttle basically needed to be heavily refurbished…
I’m amazed people in good faith will minimise what SpaceX has done and pretend like “no one else cares”.
NASA has basically zero capability anymore (given they just launch their only rocket) - which was completely disposed of.
Do people really think all these other programs/companies think it’s “better” to throw the rocket out?
Why are rocketlab and others working on recovery and reuse then? Why has blue origin failed to get to orbit? Surely we’ve been doing this for 40 years, what’s wrong with them?
The shuttle SRBs provided most of its thrust and they were certainly recovered, refurbished, and reused. As was the main engine, given that it was attached to the shuttle.
which signs? They launched a new feature (blue checks for $8) and had to turn it off immediately because it was bleeding money and ruining the platform and they have less ad revenue booked for next year than they had at the same time last time.
I don't think one should judge the new twitter course yet, but "well, the site is still up" is a very bad measure of success.
It's way too soon to tell. On a dramatically smaller scale my team went through a big drop in headcount. Day 1 the impact was nil. Day 10 the impact was negligible. Day 30 some minor problems were identified. But it wasn't until about Day 90 that we had our first outage and Day 270 that we had our first lengthy outage.
>Was Musk right? All signs so far are pointing to, yes.
Huh? He's been in charge for, like, two weeks. Did you think it could implode the instant the engineers received pink slips? Let's give it a year before we say he was right.
Way too soon to declare that Musk was right. I don't even think signs are pointing there. Twitter is bleeding some of its most valuable users, the content creators, to things like Mastodon. There do appear to be cracks happening at the edges. Bots and hate speech do appear to have increased.
Thing is, I think Twitter was bloated and it needed a kick in the rear. Pre-acquisition I heard the same from many I follow. How Musk has gone about it has been the problem. Ignoring his perpetual hates, he had a decent amount of goodwill the day the deal closed. Then, he squandered it with all his antics. A transparent content moderation board turns out to be a game-able Twitter poll. Blue check for all was completely missing any point. No one wants a blue check for money w/o the associated verification. Verification for all would have been awesome.
Ads quality has dropped from what I've seen. It looks like people are pulling out, albeit slowly. MBAs will be studying this, but how things are going means we may look back and see this as Twitters Yahoo/AOL moment when it sells for a few billion in a couple years.
Did Musk fire the right people? Or did he slash based on obedience, mood, whim? In what situation would one want THE senior SRE for the cache systems (which are unique and affect twitter across the board) GONE? There's a reason why managers were trying to claw back some engineers and employees after the big layoff. Sure you chop off a few fingers and keep working.... but lose a thumb?
What do you mean by "right"? He grossly overpaid for a business that isn't profitable and likely made it even less profitable. Even if it limps along and some husk of Twitter survives, it's hard to see Musk as making the right move here.
I guess you were also declaring Putin a tactical genius for reaching Kyiv so quickly after invading Ukraine? Military historians will be studying that advance for years.
And so is the cache setup. It’s permanently (and deliberately) running at less than 50% utilization to prevent an issue that comes up only once every 5 years (according to the author).
Of course its all bloat. Software runs on computers, not engineers. The default assumption for software is that it will go on working. The state might devolve, but the software is exactly as reliable right now as it was before. I'm no friend of Elon, and I think its hilarious to think that he can be king of twitter, but all these people talking about "code entropy" are certifiably insane.
Big tech maintains talent so that they won't use their knowledge of the system to produce an identical competitor without the technical debt or investor liability of the original.
I read it as the exact opposite: the reason Twitter is still ok* is not because all these people were just browsing reddit at work. You can't just gut Twitter to run on only a couple of hundred people and still expect the same results in the longterm.
Twitter was not a leftist commie welfare company as Musk and its fans want it to be. It was actually the fine work SRE (amongst others) put into it that makes it still tick along as it does...for now.
* actually some things are already breaking, but it will take some time for the real damage to surface on a technical level
And seriously, the gaming industry is infamous for its "year round crunch mode". It can be done. Many sectors are "hard work" (ie. public sector, fundamentally non-tech gigacorps with no real in-house IT competence, etc.)
Twitter was an innovator. I remember getting into Scala party because of some open source code library from them back then, I don't know, probably around Scala 2.8.
That was absolutely unnecessary for their core business. All of this can be done in bad C++, or plain old Java 7, or whatever, for low cost, built in Bangalore.
Now that would of course likely not withstand this kind of change storm that Elon caused. But that's a different story. (Also what Elon probably doesn't understand on a technical level (but likely guessed from a business aspect), is how flexible Twitter is. In this regard he hit arrived at the right time. Twitter is fundamentally the same as it was, now it's time to innovate with it, and the architecture he found is exactly the right one for that.)
I guess different definitions for "bloat" but how is it bloat to have a tiny team taken care of a fundamental piece of infrastructure? if the team is now gone, an issue there would mean hours of downtime. If that's acceptable then yes, it was "bloat".
I’m suspicious that most of the value in these systems comes from a small fraction of the effort and many technology jobs boil down to knowing you’re a huge cost center and putting on a performance to hide that.
This, in my reflection, is that one insightful comment that should be higher up even it came to late. Twitter userbase did not expand significantly in the last couple of years. Revenue increased. Why did cost increase so much?
Of course it was bloat. This whole “twitter is going to crash and burn” thing is a weird fantasy. Most likely it will just be run more efficiently by far less people.
Well, WhatsApp had ~50 employees and Instagram around ~15 when FB acquired them, and they were around the same order of magnitude of complexity as Twitter.
The only concern Id have is that by having so many people, your design probably comes to rely on them whereas a smaller team would be forced to make the system easier to maintain.
Personally, if I were Elon, I’d build an entirely new backend and point the clients to that rather than trying to incrementally improve what they have.
Get 50-100 10x engineers that are loyal to Elon, with big equity stakes, and crush it
A lot of people who don't understand why Twitter owns two datacenters point to "complexity" as an argument and completely disregard scale. It turns out that massive scale adds a lot of complexity to a system, particularly around many-to-many pubsub systems (like social media). It also means a lot of features, like compliance with government regulations around the world.
WhatsApp had a Billion users and 50 employees. You can say that Twitter is incrementally more complex due to wider broadcasting, but it’s also 1/5 the user count.
Everybody stating this number of employees is necessary to maintain a Twitter scale system is simply wrong.
And as technology progresses, fewer and fewer people are needed to maintain the same size system
Many just don’t want to believe that the leverage in employees favor in the tech sector is fading fast, and their labor is not going to warrant 500k comp with marginal effort anymore
Whatsapp, discord, telegram, and all other chat apps have a very easy time scaling. Using user count as a metric and comparing them to Twitter is pretty disingenuous. Instagram is a much better comparison, which had 1/50th the user count of Twitter at its acquisition.
Chat apps are well known to rely on infrastructure that scales up exceedingly well. That is why there are so many of them, and why they all have 100 tech employees or less.
Instagram at the time of the acquisition could have run on 20 servers plus S3. Today's Instagram, along with today's Twitter, does a lot of work that is super-linear in user count, and has something like 2000 engineers. Timeline building is reportedly O(n^2) in user count. The scale difference has a huge effect.
Notice that chat apps have no equivalent to "timeline building." The worst scale factor a chat app has is linear.
Ok, multiply instagrams employees at time of acquisition by 50x and you still get an order of magnitude smaller employee base than Twitter.
And obviously employee count shouldn’t scale linearly with user count. No idea where you get that idea?
To argue that a 1B+ user app of moderate complexity can’t be run by a few hundred employees is simply wrong.
The end game for all web software is going to be managed services with automation built in such that it requires almost no manual intervention (absent logic bugs). This gets easier and easier to accomplish as cloud capabilities grow. And cheaper.
The long term trend for tech is obviously towards fewer employees and bigger impact per employee through leveraging higher level abstractions. Look at what you can build as an individual today vs 20 years ago and it’s quite obvious. Unless you think technological progress will suddenly halt
You are missing that you actually pay a lot for those abstractions. Once you get to Twitter's or Instagram's scale, it starts to pay to DIY the foundations of your technology, because the people to build it are cheaper than the cost to buy it. Building also has the benefit of aligning your concerns across abstraction layers, which pays huge dividends in efficiency.
I don't think it's obvious at all that the long term trend is in favor of few employees with "high impact through abstraction" when the highest-impact people I know at big tech companies are ones who build the underlying technologies behind those abstractions. They routinely bring in 8-figure cost savings per engineer per year compared to using OTS or open-source technologies.
Headcount should scale with actual complexity of the problem being solved, which is not AT ALL linked to a user's perception of "complexity." Twitter is far more complex than you are giving it credit for, and a big part of that complexity comes from its scale and the fact that it is cheaper for them to eschew "higher level abstractions" than to use them.
Everything that is happening with Twitter proves Musk is yet another wealthy idiot who doenst know shit about shit, except how to blow his own horn. Musk is simlpy lucky that Twitter used to have such excellent engineers like this SRE so that the site isnt yet on fire.
Musk lead teams at the only company who can land commercial rockets and the first company to profit from making electric cars. To say he is a wealthy idiot says more about you than it does about Musk.
So Musk hired the right people. I mean he made some brilliant decisions over the years. Does that mean that everything he does is a guaranteed success? There are plenty examples of ultra successful people in history who were blinded by their own success and became a little bit unhinged over time..
At this point Musk is behaving just like a twelve year old’s idea of a mega billionaire. Hyperloop, flamethrowers, self driving cars now Twitter... It’s great that he’s having fun I don’t think there is much to it than that.
I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is
"the good times are over." Things start to become brittle and/or not get the attention needed, contractors are brought in because they are cheaper and/or bits get outsourced to the cheapest bidder... the professional care and attention like the author clearly brought just starts to shift over time. Consultants like me are brought in to diagnose what's wrong - the good staff could write our briefs, they know what's going on - and generally we slap a band-aid on the problem because management just wants to squeeze whatever value they can out of the assets rather than actually improve anything.
The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wholly unqualified interviewers, but they now also have to handle nightmare features that were built by people completely disconnected from the other side.
I'm not a friend of Elon's, but outside of the flashiness of the whole thing, I don't think his firing spree was wholly unwarranted.
The other day I saw a video of a bunch of people at twitter leaving that have been there for a decade or so. I mean wholly crap, this reminds me of old German industry where people retire in the place they started.
Except in these firing sprees it's not competent employees that remain - those people are usually the first to jump ship because they have options and see where it's headed. It's not like they get raises for staying to get a part of the savings from the cuts - usually they get pay freezes.
Restructuring is done on whatever idea the new owner(s) have in plan - which could be equally disconnected from "real product".
I'm not that optimistic about Twitter long term - never was TBH but this Musk thing is turning to a shitshow and also exposes what it's like working for him in his other companies.
You realise how annoying it is when you see people around you in the company making as much money as you and doing nothing useful to actually generate revenue. Data scientists that write a blog post about the most used hashtag or emoticon. Or PMs that we don't need...
>You realise how annoying it is when you see people around you in the company making as much money as you and doing nothing useful to actually generate revenue.
Absolutely. I also realize that when a company reaches the firing squad stage it's not a good idea to stick around - better polish up the CV and move to greener pastures. I have no insider information - they could be offering generous (and credible) retention bonuses for critical staff, but I've seen this happen twice before indirectly (I was gone before it started happening) and the people that stayed didn't have a good time, were lied to and left in the end anyway in a worse situation (or waited trough to bankruptcy and had to sue for outstanding wages in first place I worked in)
I have not been impressed with the average tech person's ability to understand parts of a business outside their immediate responsibilities. There were people on here seriously questioning the need for a general counsel for a company that's constantly under legal assault by every country on the planet. Never mind the long recurring "why does an ad-dependent company need people to manage relationships with the big spenders who expect that?" discussions.
By being a free speech platform without the censoring role of an editor, Twitter does not need to care if Qatar is mad about what is written there. If Qatar bans twitter, they ban twitter.
What is a "legal assault" and why does Lichtenstein have an ongoing "legal assault" against twitter?
The previous point is very reasonable, and I think your response is overly flippant and confrontational.
I've had coworkers hounded by managers from unrelated teams because my coworker wasn't working fast enough on a feature that was important to their team. They made assumptions about how many different tasks my coworker might have had assigned to them.
They were juggling like 3 or 4 fairly important features/fixes for various parts of the organization at the time because our team was (in my opinion) understaffed.
But of course, each of the relevant managers can only see "didn't satisfactorily complete my fix in the expected timeframe", and then perf review comes back and collates that info to "doesn't complete tasks on schedule" without taking a step back and recognizing that this is a failure mode of planning where one engineer was responsible for way too much stuff with no reasonable ability to push back.
At the same time, of course, engineers were getting promotions for doing exactly one thing well, which is as it should be.
There are lots of lessons in here, one of which is very likely "push back on work that causes you to overburden yourself", but I'd argue another important lesson is that you just don't see what others are working on. I'd be surprised if there were more than one or two people in your org who knew every task you've completed in the last 30 days, and to me it seems unreasonable to expect that you can make that assessment of more than 5ish people in your immediate radius.
But while we're throwing ad hominems for the fun of it; I'd rather have a bunch of "bloaters" on my team than the one person who is convinced that they are one of the golden few producing value, and everyone else is a lazy freeloader. I've worked with that type of person before and it's awful. It is rarely the case that they are _actually_ producing an 2-3x the value of other engineers, it's much more likely that they're just reducing morale and causing internal conflict while building some small piece of the pie and assuming that's the majority of the work.
Like someone who builds a button that says "delete my account" and says "I built the delete account feature" while not recognizing that there are a ton of people on the database teams that made the feature possible without performance hiccups or leaving dangling foreign keys.
You are bringing up a completely different problem that's a whole another case than bloat: bad managers that don't realise that some workers are juggling many problems, not just their own. That is very different to people doing a lot of work in secret in an unsecretive organisation, like twitter.
> I'd be surprised if there were more than one or two people in your org who knew every task you've completed in the last 30 days, and to me it seems unreasonable to expect that you can make that assessment of more than 5ish people in your immediate radius.
This is not the case at all in many work situations, a prime example being when you work on a team and all code and work produced is shared. None works on other teams. The code reviewer and hopefully project lead will know, or have a very good hunch, what everyone is producing.
I would argue that bad middle management is another bloat problem, so in essence; you just made another point for how bloat is very bad for an organisation. I wonder why you are so interested in defending the all too common effect of corporate bloat in an economy where its calculated that a large percentage of jobs are useless and there is a trend to collect multiple useless jobs.
Yes and no. I think we'd all agree that large tech companies have tons of really obvious staffing inefficiencies. There are many teams working on what are essentially vanity projects and many other teams have more staffing than they realistically need.
On the other hand the slash and burn Elon approach seems objectively terrible. Indiscriminately firing most of the company kills morale and is likely to send the company into a hiring death spiral where your good employees leave and you can't attract good talent. This won't automatically kill the product or the company but it's not going to lend itself to big positive successes in the future.
All companies I’ve worked at, startup to FAANG, have tried their best to reduce attrition and retain employees by providing growth opportunities.
Changing jobs is a skill/knowledge net-negative for the employer, and can be negative (outside of salary) for the employee as they have to relearn everything (relationships, tech debt, processes) etc.
Depends on if you want innovation or stability. For innovation, people building fiefdoms over decades of political maneuvering is terribly destructive to change.
Very few industries require constant learning for the business to compete so a highly tenured employee likely hasn’t learned anything new beyond minute process changes for 10+ years. Once people are ossified into a role like that, they will do anything and everything to shut down anything that has a whiff of threatening their current role.
That is actually not my experience. I worked at a scientific institute on space cameras, and since projects easily last 10 years from design to launch, we have a lot of older people.
Some were resisting change. Some were embracing it. All in all, I had a great time and gained a lot of respect. I love working in teams with all ages.
Your sentence "once people are ossified..." feels like a generalization to me.
> I mean wholly crap, this reminds me of old German industry where people retire in the place they started.
German here. I think this actually is a huge part of the success of the famous Mittelstand - all that institutional knowledge these people have is extremely valuable. It's not just basic stuff like "know time tracking, billing and other admin systems and internal processes", but also the stuff that really can speed up your work: whom to ask on the "kurzer Dienstweg" aka short-circuitting bureaucracy when needed, personal relationships with people in other departments on whose knowledge you rely (it's one thing if you get a random email asking for some shit from someone you don't know, but I'll always find some minutes to help out someone who has helped me out in the past), all the domain-specific knowledge about the precise needs and desires of your customers...
Attrition is bad for a company as a whole, the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact), and so there is no KPI for leadership other than attrition rate itself.
The only problem is that over the last years, employers' mindset has shifted from regular wage raises to paying the bare minimum which makes changing jobs every few years a virtual requirement for employees to get raises, and so we are already seeing the first glimpses of US employment culture and its issues cropping up.
> the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact)
This is false. People invest on many year horizons all of the time when they believe in the company. If there was good evidence that having an average tenure of 5+ years was a great boost to the company, people would clamor to invest in companies like that.
This was basically Google’s entire pitch when they were an early public company. Happy employees == great things. I remember when 60 minutes or dateline did some special about google before 2010 and people were floored by how good the employees got it. However, hiring standards relaxed and now google is slowly rotting from the inside to realize it’s the new IBM.
Long tenure alone is absolutely not an advantage. It’s very easy to have a bunch of dead weight that just looks like they know what’s going on.
So back to your point about experience mattering. That should show up in customer satisfaction, project turnaround speed, success rates, etc. Otherwise it’s pretty meaningless. And guess who already measures those KPIs…
> This is false. People invest on many year horizons all of the time when they believe in the company.
The perfect example of why that is not the case at least in Silicon Valley is Google and their messengers and social networks... just how many of them exist(ed)? Four? Five? Each of them was someone's promotion project, left to languish, suffer and eventually die afterwards. Honestly it's a wonder Gmail and Maps are still around, their UI hasn't been updated much in ages despite obvious potential for improvements... I guess the only reason why they are alive is that they generate absurd amounts of data.
> This is false. People invest on many year horizons all of the time when they believe in the company.
Granted, but they usually have the option to sell at many points along the way.
Think about an alternative financial instrument. What if you could invest $X in a public company (and get a higher rate of return because of the higher risk) but you lose the ability to sell until five years have passed?
This is one kind of option that might be worth trying. I want to see more mechanisms promoting long-term investment in our public markets. (Bonds are not identical to what I described BTW.)
The lack of these kinds of options (sure, bonds exist but I've not noticed them used with any notable significance in public companies) is why people talk about the short term quarterly focus of Wall Street.
I don’t understand what the point is. You have the option already to hold on for 5 years+. Many people do this (especially employees with vesting schedules) and they are rewarded for doing it.
> is why people talk about the short term quarterly focus of Wall Street.
This is a false meme though. Tesla was worth more than multiple profitable long standing car manufacturers combined before it ever turned a profit. Investors fixate on short term performance when the company has no long term vision and deriving cash flows is much easier (e.g. a container ship holding company).
> You have the option already to hold on for 5 years+.
The difference is having the option. :) Behavior tends to be different if you don't have the option to sell during a window of time.
In this hypothetical case it might serve as a built-in grace period. Companies that otherwise might have failed if measured and punished quarter by quarter might have time to build a product over five year time horizon.
> Long tenure alone is absolutely not an advantage.
It's also not necessarily a disadvantage. The real argument is that people management is important. Everyone from the new person who just joined to the decade long senior engineer need to be held to high standards. Which brings me to my next point.
The big problem with Twitter was management. Dorsey was barely a CEO for many years. Parag didn't seem to want to try anything. I think people routinely underestimate how important good leaders are. "There are no bad teams, only bad leaders" comes to mind. So far Elon has shown himself to be terrible in this regard. With SpaceX and Tesla he communicated a vision, with Twitter he's communicated very little and just instituted chaos.
> The big problem with Twitter was management. Dorsey was barely a CEO for many years. Parag didn't seem to want to try anything.
The problem was even worse: there was no vision at all from no one, not leadership, not the investors, not the users, what Twitter should be, other than "it is a way for instant communication with feeds". Everything else was completely lacking: what features do people want, what moderation policy should be applied, how does Twitter plan to make money.
The only ones that had at least some sort of vision where activists - the left wing, the advertisers and large parts of the users didn't want Nazis any more, and the right wing wanted "free speech" aka allowing Nazis.
I think this is down to the time horizons each kind of company optimizes for. Mittelstands are usually owned by families and they basically try to build a business that will feed the parents as well as kids when they come of age. Publicly listed companies usually optimize for this quarter. So even if you are looking at the same KPIs, you could get different outcomes just due to the utility functions being different.
> The reality is most huge companies are majority bloat.
This is true, and in my opinion, true for a reason. And that reason is not "most huge companies are dumb", as opposed to what Musk's cult seem to believe. The reality is, measuring what exactly is "bloat" and precisely cutting that bloat is extremely difficult and firing more than half of your workforce is probably like using a warhammer to do brain surgery.
Well, your alias actually refers to that process. Large organizations gain entropy over time. And entropy is precisly the inability to describe some structure at a micro/nano-level.
But while fighting the entropy can be hard, not doing so is almost certain to be lethal.
If an organism is infected by gangrene or cancer (the more extreme forms of entropy for a body), it may be more realistic to cut away whole body parts than to treat it in place, even if there is risk of sudden death.
It seems to em that this is what Musk is trying. Either Twitter will be gone within a year, or they may very well be sustainably profitable within 2-3 years.
Software engineers at Twitter got used to working on an money losing company for 10 years and being told they’re great at their job. Then they were fired en masse because someone’s called them out on it.
If your company is losing money all this time you are likely to be fired eventually in the real world. Job security in sw world had become so high that no one seemed to expect it. Everyone assumed “sure we’re losing money and the company has no direction” but all is fine.
They all stayed there in their tables working for 10 years in a rudderless company as if it was a government job.
This is factually untrue though. Twitter was making money in 2018 and 2019 (to the tune of ~1.2B/year in net profit out of ~3B revenue, which is a fairly high profit margin) they lost quite a bit of money in 2020 and less in the years thereafter. However, even in the years where they had negative net income EBITDA remained positive suggesting the losses were upfront investments that would be expected to be amortized over the coming years.
The only reason Twitter is in deep financial shit right now is because Elon acquired it in a leveraged buyout and the cost of servicing the debt is estimated around 1B/year.
Yeah, most of the complaints of Twitter was that it should be doing more in the space it commanded, not that it was losing tons of money every year. Though, 1-1.5B in yearly interest payments is going to make that tough going forward.
>The reality is most huge companies are majority bloat.
Something I learned early in my career is that some companies consider this a feature not a bug. As in, they are hoarding talent so that they don't go to work for their competitor.
Yeah, but that's not what's happening here, is it? Musk is pretty obviously looking for a corporate culture shift, not turning the company into a cash cow - and for this to work, entropy needs only to work slower than half a year. Which this article argues pretty convincingly is going to happen.
I don't know man. His last pure software company was PayPal and they fired his ass from the CEO role and put in a competent CEO (Peter Theil) to right the ship and sell it off before the 01 crash.
Tesla and SpaceX have been wildly successful and he will forever recieve some well deserved credit for their successes but at the same time I think its pretty bad that Tesla (and SpaceX) have had multiple near death experiences and some them could have been avoided if it weren't for incompetence.
I followed the Model 3 production religiously and the amount of dumb mistakes they made even after other industry experts were yelling at them about imminent iceberg collision is just unbelievable. In fact the Model 3 production, with its horribly designed body that he wasn't keeping on top of and Musk's stupid attempts to automate all of assembly should be entirely attributed to him. It was so bad it was worse than even the shortseller's worst predictions.
I guess you can give him a pass for near bankruptcy #1(Roadster roll out) and #4(COVID lockdowns). Not sure about #2(Model S early sales woes and potential acquisition by Google)
All I am saying is that he fucks up A LOT and has thus far managed to stave off the consequences of his decisions but many occurrences have been down to luck and luck is something you should never bet on.
The US has been formed religiously by various congregations that descend from Calvinism, which teaches that hard work earns you the grace of God. In modern times, this has morphed into Prosperity Gospel which teaches that if you are loved by God, you will become materially successful, as well as the flipside; if you are materially successful, you must therefore be loved by God.
The billionaire worship is simply the secular version of this faith. X made billions of dollars by owning a business that does Y, therefore X must be an expert on Y, and businesses in general, and is a genius, and is a morally superior person. Otherwise X wouldn't be a billionaire, because - horror of all horrors - the world is just, isn't it? It couldn't be the case that X just got lucky, can it? Surely, X has earned their billions, right?
You’re still fixating on the wealth part, which people on this site (at least the arguments I see upvoted) do not.
People are “worshipping” Musk because of his repeated success in making successful large innovative businesses. The wealth he has is a product of that, but the deference is to the success of the companies.
Any optimization he might do is unlikely to offset the cost of servicing the debt he offloaded to Twitter's balance sheet after the acquisition.
So at 5% interest the yearly cost is 650 million, even at some obscene and unrealistic payroll per employee (including not only developers), like let's say 400k he'd have to fire over 1600 people just to balance this out. There is no way to spin this positively. Also due to no more RSUs cash comp for remaining employee would need to increase.
Add to that the the loss in advertisement revenue do to obvious reasons.
Taking these facts into account could you please explain how is Musk NOT a liability to Twitter? He basically spent 31 billion (plus 13 billion he offload to Twitter) just to buy the trademark and the user base (and some proportion of advertisers...). From a financial standpoint that's an objectively terrible deal. Musk might have other goals and that's great but in no way this is comparable to Tesla or SpaceX.
>But he did change the world with his companies already, didn't he?
History is full of examples of extraordinary successful people who get blinded by their own success and start thinking that they are infallible. This ussually results which in pretty unhinged behavior when they get older. Musk does not seem like the most grounded person ever so he's likely at a much higher risk of this than almost every tech billionaire I can think of.
also: History is full of examples of extraordinary successful people [period].
Maybe somebody with Musk track record can be seen as more credible than some random "qwytw" on the internet? I mean incentives are on his side, aren't they?
For one his famous. It's his toy, so he won't let it disappear into irrelevance without fighting for it. There are not many people currently alive with comparable track record, being recognized world wide, with deep pockets like him.
What if in few years time from now people will be stepping on each other writing posts and books on how they really knew all of that from the beginning, that obviously he was right etc. - doesn't sound unreal or alien to me, quite opposite actually.
To be honest, I don't even think anything extraordinary needs to happen for all of it to work out just fine.
For all people saying technically it's not just rendering short messages - well it is just that at the end.
Look at stackoverflow - most people don't realize it's just single monolith application running on 6 servers - all of it.
There are still thousands of people employed there, which feels like too many if anything.
Did I misrepresent any facts or figures? Did I lie? What does credibility have anything to do with anything otherwise? It's an objectively bad deal from a financial perspective, if you have any arguments against that you're free to share them.
Twitter might do "fine" as a platform. Sure, why not... Does not change the fact that he grossly overpaid and that the 13 billion in additional debt will be financially crippling for the company. He might sell some more Tesla stock to offset that. I mean it's his money, he can do whatever he wants (unless the SEC stops being a complete joke).
> that obviously he was right etc. - doesn't sound unreal or alien to me, quite opposite actually.
Right about what? Even Zuckerberg seems to have a clearer vision about the Metaverse than Musk seems to have about the future of Twitter. I might be wrong. Care to enlighten me instead of talking about some random tangentially related things?
> There are still thousands of people employed there, which feels like too many if anything.
Sure, can't really disagree. However is it better for a business to have an additional 2000 employees or so who are possibly superfluous or to fire all of them and spend ~600 million or thereabouts servicing a debt incurred for no reason? It's a pretty straightforward you chose to ignore for some reason.
And that makes him infallible? If anyone else but Musk bought Twitter and started doing what he did they'd be the laughing stock of the financial world. His reputation is the only thing keeping him afloat (in this specific case, he might still do great with SpaceX and Tesla but so what? People like Jobs, Buffet, Gates etc. also made significant blunders at some points in their career)
To play devil's advocate for a bit, are you certain he's solely responsible for the successes of SpaceX/Tesla, or that those companies succeeded in spite of his leadership?
In case it isn't clear, I agree with the parent comment. In the same tone as the comment above the parent, you could also write "Thing is - Elon Musk knows a thing or two about sexual harassment lawsuits. It's inevitable that he will have another at Twitter."
Stop idolizing billionaires.
>Musk has successfully run companies in the past and is running two extremely successful companies right now.
Is he actually running those companies, or is he posting selfies of whiteboards at 1:30am showing architecture diagrams of Twitter?
In these markets, after he was nearly forced by courts into completing a purchase deal for Twitter after making an overpriced offer, I'm genuinely curious to see how he pays the ~$1b in interest per year on the debt to purchase Twitter.
No, people are idolizing Musk because he is a billionaire.
People who obviously know absolutely fucking nothing about coding or manufacturing or running businesses or software engineering or cars or space or rockets or emerald mines are continuously defending Musk, and their only justification is that "he's rich, so he's obviously smarter than you!"
> No, people are idolizing Musk because he is a billionaire.
No they are not. They are idolizing him specifically because of the success of Tesla, SpaceX, and PayPal.
> "he's rich, so he's obviously smarter than you!"
Nobody is making that argument. Its usually, “look at the success of the companies he leads”.
You can make an argument about whether or not he should get the credit he does for the success, but nobody is idolizing the wealth.
None of these people are going to defend the great insight of the winner of the last powerball because there is no repeated pattern of long term success.
Look at his claim that Twitter is slow in India because of poorly batched backend calls [1].
He's simply regurgitating what he thinks he heard from others without really understanding how anything works. He knows less about software architecture than an intern picked at random.
Stop repeating this, because Elon had the gist of it correct, if not the specifics.
a) Twitter is atrociously slow for something that displays a mere kilobyte of text at a time.
b) Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network.
c) Most of the rest is because of a thousand microservices chatting away at each other. Not technically client-to-server, but still taking significant time.
Elon accidentally conflated the network path of (b) with the performance issues caused by (c).
They're both bad! They both need fixing! The fact that Musk mixed up doesn't matter.
> Much of that is because of hundreds of client-server JavaScript requests on first load.
Isn't this not true though? Based on the responses to his Tweet only one GraphQL request is made which batches the whole thing.
> Most of the rest is because of a thousand microservices chatting away at each other. Not technically client-to-server, but still taking significant time.
But what does this have to do with India? If the slowness is due to calls between microservices hosted on (from my understanding) their own hardware, why would it matter if the initial GraphQL request comes in from India, since all the rest of the backend requests would be the same?
> his Tweet only one GraphQL request is made which batches the whole thing.
Literally bang in the middle of an F12 network trace showing dozens of requests, mostly JavaScript. Not JSON RPC, per se, but still an overhead taking up seconds, especially on low-end Android devices.
> But what does this have to do with India?
Everything. Nitter uses 40KB of code & data to display the same content that takes Twitter megabytes.
People in subsequent threads have pointed to the gigabytes of "data" usage by the Twitter Android app, which is just absurd for an app that displays primarily text.
It's a bloated pig of an app that performs poorly on 5G networks in the United States, and hence terribly on slow and overloaded networks separated from Twitter's data centres by an ocean.
"It works fine for me" says the engineer connected to the WiFi in Twitter HQ.
> Literally bang in the middle of an F12 network trace showing dozens of requests, mostly JavaScript. Not JSON RPC, per se, but still an overhead taking up seconds, especially on low-end Android devices.
I'm pretty sure that's my point? How are you equating "dozens" to "1200" (or even "hundreds" in your last comment) as though Elon is in the right ballpark?
As for the rest, I don't know what your point is, where did Elon say anything in that thread about how much data they were sending? He's quite literally responding to someone saying it was the amount of JavaScript being downloaded and telling them they're wrong, it's the "1200 RPC calls". I'm pretty sure the point he was attempting to make is that it's slow due to thousands of serial "RPC" calls made between the app and Twitter (which will have a higher RRT to India) but as you've demonstrated that's not what it's doing.
I'm not saying Twitter is well designed, I'm just not pretending like Elon somehow had the right idea when you've had to stretch it this far. If he had a proper technical understanding then we wouldn't have to have such a discussion of what he "actually meant".
That's correct, I am working in mobile world since 2005 and the latency of mobile networks is the most bad thing when you build mobile apps. The more requests you do the more latency you introduce to the user, especially if it's fetching data for UI, even if you do it asynchronously you have to wait for the data to update the UI.
> Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network.
This is not true at all. I have a look and it's not that there are many REST requests it's that the requests send back a load of data. For Example the timeline sends back a 1MB JSON response. It's heavily gzipped so comes in at 100KB over the network but it takes quite a while for the whole request.
In places like india the internet is slow and many people are using VPNs. I emulated this and I can see that over VPN on a slow internet connection it's like 10 seconds to load the home page.
He was right on the details of the two different issues, because those were explained to him.
But he conflated two things that he didn't really understand and reached a completely wrong conclusion. He's shared that conclusion with confidence and never admitted he was wrong. That's a fatal flaw for someone who holds a Chief Engineer title at a rocket company.
He's a CEO who is has been incredibly successful at selling a vision and getting people to do what he wants. He is not an engineer. He should stop role playing as one.
He also doesn't make decisions like this from monologues in his head - he surrounds himself with people having deep expertise and makes bold decisions based on feedback; if something doesn't make sense, they'll stop doing it the next morning. In "normal" corp not only it'll be there 3 years from now, it'll also grow fat around it and spawn other idiotic initiatives with little to no (negative?) value.
SpaceX is subsidised (and so is Tesla) and automotive industry was already in the process of electrifying. Tesla just put pressure to speed it up. Haven't checked but it would not surprises me if Formula E predated Tesla.
Winning government contracts over other competitors is not the same thing as “subsidized”. Otherwise Hilton is subsidized because government employees stay there sometimes.
I get the impression that some government contracts are used as stealth subsidies. And SpaceX, Tesla, and Starlink have benefited from plenty of actual subsidies, too.
To be clear, I'm not implying that's a bad thing. These companies seem like an example of subsidies working as intended, i.e. driving innovation.
Can you give specific examples? Diluting the meaning down to “the government is also one of this company’s customers” just makes it meaningless. That will cover nearly every large company.
In January 2020, Toyota passed the milestone of more than 15 million self-charging hybrid vehicles
If every vehicle sold by Tesla since its inception up to the most recent Tesla sales figures released for Q1 of 2022 are accounted for, there are over 2,645,000 Teslas on the road worldwide
Sure, cheap hybrids. Nobody is buying them because it's their dream car though, it is just affordable and cheap to run. Tesla makes cars that are desirable.
Funny enough it would be hard to find anyone with more CEO experience than Musk. This doesn't mean he does all the management, often the work of these guys is to find the right people to do the management for them.
Yeah he is so smart, he locked himself out in firing spree during some manic phase and then had to beg people he just fired to let him in. If anything, this twitter saga showed serious cracks (that were there before if you looked closely) in his persona. Vengeful, petty childish behavior was there before (ie Musk accusing main Thai cave rescuer for being a pedophile, because he refused his submarine concept? Plenty of similar stories)
Its a matter of approach - if you focus on all his success, you can continue worshiping him as some capitalist messiah. If you focus on bad stuff, he is a fucked up mess to be polite, a bad parent and husband, an even worse human being in general, utterly horrible boss, person with deep mental issues that aren't clearly tackled in appropriate ways.
The truth, somewhere in the middle of those 2 extremes, still ain't that nice IMHO. He burned basically all good PR capital be built over the years, basically nobody sees him now as good person, just another ultra rich a-hole like Bezos that needs to be kept in check. As he will find in following years, people's opinions trickle down to cash flow. If I would be buying an electric car right now (which doesn't make much sense for another decade to me), Tesla would be off the table for example.
Literally his first two companies, Zip2 (CTO) and x.com (CEO), were pure software companies?
Another company he founded, SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do.
Another company he runs, Tesla, is famous for being one of the few car companies that makes software for their cars that people actually want to use.
If a citation is needed, it is a citation to explain how anyone could possibly believe with that track record that Elon Musk doesn't know how software is built.
Knowing how to run a software company and knowing how to build software are two things. The first thing usually implies hiring the right people for the second thing. I’m sure Musk knows at least a bit of that, but everything we see with Twitter is that he’s axing those people at a rate that you can’t keep up hiring at. So there is some obvious disconnect here and I don’t think “Ah, he knows what he is doing “ is a sufficient explanation.
The Zip2 application ran on a single PC. Technology has changed a lot from the early Java days(I think what Zip2 was written in). Perhaps this helps explain some of his public mistakes in understanding the codebase of Twitter.
> SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do.
Not to be dismissive of landing an orbital rocket, but that's mostly a feat of aerospace engineering and control theory. On the software engineering side of things for SpaceX I think their work around CFD would be the more impressive feat from a computer science perspective.
Elon just made sweeping changes that cratered their ad revenue and took an axe to engineering teams all over the company. He did this in the lead up to the holiday season which has been the most profitable part of the year with advertisers pouring a significant portion of their yearly ad budgets into ad buys. Even if he was 100% correct that all of this was bloat, it was a monumentally stupid idea to make these changes now of all times. I would have expected him to at least have enough of an understanding of software engineering to know that deprecating systems and cutting out bloat needs to happen carefully with the talent onhand to be able to back out changes and manage the transition seamlessly. Fundamentally the risks involved are more financially impactful than the price of keeping headcount around for another year to do it safely.
Twitter hasn't even had a major incident yet and just the revenue loss from advertisers pausing ad campaigns probably already offsets all the savings they were hoping for in 4 - 6 months once the layoffs finally start bringing down staff costs.
Look, I actually do agree that Gwynne Shotwell has been a key component in making SpaceX successful / curbing Musk's worst impulses / etc.
But don't you think it's strange that (assuming so much of SpaceX's success is owed to Shotwell) nothing like what SpaceX has achieved was achieved at her previous gigs? But then she goes to work for Elon Musk and suddenly...
Regardless, the point I'm making is not "Elon Musk wrote all the software at his various companies himself" but rather "Elon Musk founding / running / exiting a variety of companies in very diverse fields but all of which are good at software is a very strong indicator that he understands software and how it is built".
Meanwhile, the "he doesn't understand software" corpus of evidence basically amounts to "I think he's doing a bad job of running Twitter in the handful of weeks he has been there. I think this even though I have very limited insight into what is actually happening at Twitter, besides what is being fed to the media by disgruntled employees who probably hated Elon Musk before the acquisition and their subsequent firing".
I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life.
In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's of devs who are not 100% aware of what's there already can't figure it out. I've taken over systems that I knew very little of and were pretty big (not as big as twitter) and I managed for years to make changes without drastically breaking stuff. In any event if they do break stuff they will be able to fix what they have broken.
No, the real failure here is the massive debt burden and the fact that there is no way that twitter can ever service that amount of debt. Note that before EM took over it was ticking along with a relatively small loss. If they had cut headcount by maybe 10% they would have been break even easily. There is no way that's possible with $4million of interest per day. They have to radically change the way they monetise the platform to get to that level. I don't think they will ever get there and Musk will sell off at a bargain basement price at some point in the future to pay back the debt.
The fact that he was able to "buy" Twitter and yet transfer a significant amount of debt to the company rather than being liable himself is just another sign of how different "rich people accounting" is. He will walk away from this with a bit less theoretical money but no material impact to his life, while thousands of people are having their lives up-ended. How long are we going to keep letting shit like this happen?
Yeah I love it. I would like to "buy" a multi-million dollar mansion in the Bahamas with big bank money, but have the bank only encumber the property itself, and I get to live in it and pay service staff for my luxury, maybe the odd AirBnB letting to maintain the occasional pretense of repayment, up until the time 'it' can't sustain 'itself' and I walk away debt-free.
But apparently I'm not a rich person so that kind of accounting doesn't apply to me.
True, but the difference is all about what happens when you can't pay the 10% interest or get into significant negative equity. If you can just walk away, then you have "rich person accounting."
Fyi, if you live in the US, you might actually have access to that kind of financing. Non-recourse loans for retail clients do exist in many states in the US, including California and Texas, and they're the default for all government-backed mortgages in the US [0]. But globablly speaking, they're the exception, yes. I don't know what the situation in the Bahamas is like.
Leveraged buyouts are certainly a thing that sounds like it shouldn't be possible the first time you hear of it. It seems extremely odd that you can buy a company with money you don't have and then have the money take on that debt instead of having to take it on yourself.
As I understand it, this works by rounding up potential loans, approaching the board of the company and getting them to sign over ownership of the company for a pittance in return for the shareholders being paid out by the company using the loans you brought in. This feels more like an emergent property of the system (specifically, contract law and how publicly traded companies operate) than how the system is intended to function.
Intuitively this shouldn't be possible as it's acting against the company's own self-interest despite being in the interest of the shareholders (and the buyer), but I think "the company's interest" in practice is defined by "the owners' interest" (and the owners in this case are the shareholders, who sell the company). I guess corporations aren't people after all.
The question is who is left holding the bag? If the stakeholders get paid, they are happy. The new owner is debt free, and have an ownership, so that's good. It seems to me that it may go two ways: either the lenders will be left holding the empty bag, if they can't make the money back - and it's their fault for doing a bad due diligence; OR the company gets sold off piece by piece to satisfy creditors (if they can't make the business work), which again a good thing - what purpose serves a company that cannot produce a profit ?
This view is sufficient if you view corporations entirely as entities that serve to create profit. But this view ignores externalities which are not part of this mechanical description. Specifically, what perceived value the corporation provides that allows it to generate that profit. In Twitter's case its monetization has primarily been ads, so I'm referring to what makes people invested enough in the service to make it worth for advertisers to pay to show ads to these people.
But of course your view is what's reflected in law: the corporation exists to generate profit for its shareholders, so if killing and selling it for scrap (whether directly or by proxy in a leveraged buyout ending in a firesale) provides more benefit to its shareholders than keeping it operating at a small profit, that's the logical decision. The corporation is not really "a person", it's a vehicle for its stakeholders (or shareholders). A stable service puttering along without making big profits or losses is considered the bad ending.
> Intuitively this shouldn't be possible as it's acting against the company's own self-interest despite being in the interest of the shareholders (and the buyer), but I think "the company's interest" in practice is defined by "the owners' interest" (and the owners in this case are the shareholders, who sell the company). I guess corporations aren't people after all.
Or maybe they are too much like people. Right now there's a bunch of things that I should be doing, that would be in my best interest - continue with my TODO list, or do some exercises. Instead, I'm browsing HN. This shouldn't be possible, but it is, because I'm a human - what I want to do, what's in my best interest, and what I actually do are three different things, and rarely aligned.
(Ironically, in humans this is usually called an issue with executive functioning, whereas in companies, it's the reverse.)
You were not around during the period of highly leveraged take overs starting in the 80s, were you? That is classical hedge fund behavior, if done right. The fact that Musk lending banks only get around 60 cents on the dollar on the market for the loans they gave Musk shows that it might not have been done "right" when it comes to Twitter.
"Lenders that also include Bank of America, Barclays and Mitsubishi UFJ Financial Group committed to provide $13 billion of debt financing for the deal. Their losses would amount to $500 million or more if the debt were to be sold now, according to Bloomberg calculations."
There is Elon's private wealth, mostly in tesla stock from what I understand. The cash and assets from the companies at which he is (majority) owner, CEO or a combination thereof are not his, they belong to his companies, which are separate legal entities.
Yes that's the approximate amount they have to pay due to the way he's done the financing of the deal. Keep in mind that the current interest is at a similar amount to there revenues from last year.
That's the reason to deeply cut expenses and to try and make more money. He could probably have serviced that debt if everything just kept going as normal (no big ad spend cut) and he had fired everyone but that's an impossible scenario.
It’s going to be something like that. He bought it for 44 billion. That was mostly loans. Those loans are transferred to the company and the company pays interest on them.
The banks committed to the loans at the point of the takeover proposal (end of April). The macroeconomic lookout was better. Since then, just about every tech stock is down >20% and Musk has helpfully made unfounded accusations of bots and spam and whatnot in a pointless attempt to delay the acquisition, meaning the banks now have to try and unload these loans in a recession. I think last I heard they are giving a 40% discount or just keeping them on their books, hoping for a recovery as Musk trashes the company value further with every week that passes.
Yep, agree. Twitter's revenue was primarily from ads, and now I'd bet their ads revenue has dropped a huge amount. Given how Musk is behaving now (reinstating Trump etc.), and losing many pivotal sales people, and the firing of a ton of people in charge of dealing with hate speech and such, it seems unlikely advertisers will return now.
advertisers wont care in a couple months. the situation has lots of public attention now, which is what the advertisers are actually afraid of. They don't want their brand associated with the craziness. They don't actually care one way or the other about any choice musk makes. They're not going to just walk away from a 300+ million person audience permanently just due to principles
No but if there is no content moderation and Twitter becomes even more of an open cesspit than it already was, announcers are going to stay away because most of them do not want to be associated with toxicity. Perhaps politically active organizations will want to be there (e.g. the NRA or the ACLU), but nothing family-friendly (e.g. Disney) and most likely nothing Christmas-y during Christmas season.
Only if ill-intended parties pressure those advertisers and effectively threaten them with that bad publicity for sticking around. It changes the game when there is coercion by third parties who are more than happy to see Twitter collapse if it isn't governed by their desired policies and worldviews.
If Twitter becomes (even more) a platform for hate (or at least "highly militant") speech, placing ads on Twitter, especially close to said hate speech becomes bad publicity, regardless of third parties. If you ware placing an ad for a mattress or a family movie, would you like it to be seen by people who are in the middle of a flamewar?
I don't presume to be an expert on how reader sentiment colors an ad view. But it would surprise me if it had as big an impact as people claim, or if people actually think the advertiser is picking a side or condoning bigotry, for example. We are used to seeing unfitting advertising strewn across the web, for better or worse, without correlating it with the subject material of the page. It would at worst devalue the ad space for being poorly targeted.
Furthermore, you would need to supply evidence that hate speech is more prevalent instead of merely fearmongering that it is allowing such.
> I don't presume to be an expert on how reader sentiment colors an ad view. But it would surprise me if it had as big an impact as people claim, or if people actually think the advertiser is picking a side or condoning bigotry, for example. We are used to seeing unfitting advertising strewn across the web, for better or worse, without correlating it with the subject material of the page.
That's entirely possible. I'm certain that big brands know better than me.
> It would at worst devalue the ad space for being poorly targeted.
Isn't this kinda synonymous with what I was claiming? Or are you saying that big ad spenders would come back but only if Twitter lowered its pricing? It's pretty clear that Twitter will do the latter, so we'll see about the impact.
> Furthermore, you would need to supply evidence that hate speech is more prevalent instead of merely fearmongering that it is allowing such.
And yet it has happened with great regularity. YouTube alone has been the site of numerous such advertiser withdrawals, so much so that its users coined a name for them, 'Adpocalypses'. It is often journalistic enterprises who see fit to gin up the severity of the content and to target any advertiser for coercion whose ad appears on a given 'problematic' video.
At this stage, I have no clue whether there is or will be content moderation in three months time. By now, Musk has said many, many things, often contradictory. Some of these tweets may turn into facts but I doubt Musk himself knows which ones.
I stand by my earlier claim: if there is no (or not enough) content moderation, most advertisers will not return. If there is, they may.
They might if other channels aren't saturated and things are still in flux. Kanye was unbanned yesterday. Do you think he's going to behave better or worse than before? If he behaves worse then that's another headline, another news cycle about whether he should be banned or not.
Right now the trend is advertisers getting more spooked because of unpredictability and Musk being unable to reassure them in calls he has been reported to have been on. I expect this trend to continue
As I understand it, most of the ad slots were sold at a yearly sales event. This time most advertisers were cautious because the company couldn't respond to questions of how Elon Musk's takeover would affect the company (and specifically the type of content alongside which the ads would appear) as this was happening before he took over but after the sale was practically a done deal. This left them with only a fraction of the ad slots "pre-sold" compared to a normal year.
Additionally it seems a lot of advertisers have cautioned their ad departments to halt spending on Twitter ads for now because they're still unclear about wtf Elon Musk is up to (not to mention the mass impersonation fiasco that allegedly resulted in share prices of several companies being affected negatively).
It's not just sales people, Twitter seems to have gotten rid of most of its external communications people to the point where journalists can't even find anyone to ask for statements other than Elon Musk himself (which may be intentional given that he likes to present himself as the face of his companies).
But if Twitter's ad sales really did rely so much on these sales events, I'd wager that the sales people largely existed to communicate directly with long-standing big-spend advertising partners. It's hard to overstate how much these types of deals hinge on "social glue" rather than cold hard numbers (which can be interpreted/reframed to be interpreted however you want as long as both sides are happy).
Well it's all rumors, but if the news was good then Elon would be trumpeting it from the hilltops since his brand and the company's depends on the perception of his business acumen. He's certainly trying to do that[1] whereas the response has been lukewarm. Meanwhile, his people with direct relations to the major ad buyers are getting fired[2], and the handling of the relationships with industry groups seems poor[3].
The issue is Twitter under it's current debt burden can't afford to lose any of its advertisers. It needed to find a $1 billion of savings somewhere to keep the deal viable, and not lose any advertising revenue. Due to the acquisition that's off the board entirely due to tanking their guaranteed revenue[4] to the tune of $600-900 million, and since then Musk has done almost everything possible to drive away ongoing spend. So at this point, Twitter is in the hole to at least $600 million of revenue, bringing their minimum current loss for this year up to ~$1.5 billion. If you needed to explain one thing, it's that the massive quick firings were a desperate move to not lose money on that, but it's a false economy and Elon's being ruining any gains by continuing to make moves which seem like they're hemorrhaging even more ad-dollars.
If you consider that engineers cost about $1000 a day and he got rid of 4,000 of them… oddly it works out to about $4 million per day.
I’m not an expert in math but it’s seems pretty possible.
Plus they’ll have a lot more impressions to sell now that people are allowed to speak. Ad rates might drop but then someone will write an article about how they are getting great CPC on Twitter and everything will be back to normal after the blue checks have their sob fest.
Dead cat bounce. Engineers are paid $1000 a day because they create value to the tune of $10 000 a day. So maybe you saved $4 million a day in very short term, but now the momentum is gone, and soon you will not be earning $40 million a day from the value that now was not being created, just in a "few" short months.
I wish Elon the best, but he could have hired his own team of "hardcore" engineers and put them to work 80 hours a week, spending a fraction of the price that was paid.
> Engineers are paid $1000 a day because they create value to the tune of $10 000 a day.
If this was true then either the non-engineers in twitter have been destroying many times their salary in value or twitter has been obscenely profitable these last few years.
SWE are not assembly line workers, they are the people that build the automated assembly line. It’ll keep functioning well enough without them and with a much smaller team performing maintenance. I’d even bet on new feature being released faster now that they’ll be owned by highly motivated individuals instead of committees of managers and randomly assigned engs.
I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post.
Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies. As always, the last 2% is a hard problem and business doesn't really want to deal with a such unreliable platform. You wanna onboard big advertisers which potentially spend $100M ARR? Then you need to assign a dedicated account manager to handle all customer escalations. PMs then triage and plan their feature requests and later engineers implement it. Which all adds up.
And they also uses your competitor's product, like Google, FB, TikTok etc etc... Twitter is a severely underdog here, so you need to support at least a minimal, essential subset of features in those products to convince them to spend their money on Twitter. That alone takes hundreds of engineers, data scientists and PM thanks to modern ad serving stacks with massive complexity.
Yeah, it ultimately boils down into a simple fact that it's really hard to take other folk's money. You need to first earn trust from them. They want to see if your product is capable of following a modern standard of digital ad serving for now and foreseeable futures. Twitter has spent lots of time for earning trusts and the original post is one evidence of such efforts. And this usually needs more man power. You might be able to do that in a more efficient manner, but I don't think that's as simple as firing 75% of your entire headcount.
> Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies.
This exactly. During the recent Whatsapp outage, many threads popped up on HN about how big of an issue this is in Europe, since Whatsapp is the main messaging platform in Europe. Thankfully, these outages are short and far between, so they never actually cause real issues. This is obviously costing Meta/Facebook a lot of money, but allows them to be an essential service. So essential in fact, that every major news outlet in my country sends a push message as soon as Whatsapp is down.
If Twitter wants to be a comparably important platform, they need that same stability. And Twitter, for me, is very much the best place to stay up-to-date on any current event (in near real-time). Reddit used to be pretty good with Live, but that's pretty much died (and was mostly a summary of tweets anyway). I really hope Twitter survives Elon, because I don't know of an alternative right now that has the same value in this use case.
I think the opposite. Many softwares at its best when the team was small. Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth. It is not necessarily good for the product, actually many times, it hurts the product, but overall it is good for the company, the company will enter new areas, can explore new things.
What Twitter is doing is to scale down first, focus on the product, and once it gains traction, it definitely can scale up again. I don't think it will hurt the product very much.
> Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth.
I don't think you have a good understanding on how those companies are growing and scaling out. Don't take growth for the granted. "Right product" or "Right technology" won't give you that. It only comes from solving thousands of very specific, never-ending customer problems. If you do B2B, you need to spend most of your time on very specific requests from priority customers. And they are not one, but hundreds of them if you targets $xB business. It's just physically impossible to keep up with a small team even with a very aggressive prioritization.
Still not convinced? Google has a notoriously bad reputation for their customer supports and it's primarily because of their tendency of keeping "inessential headcounts" low as possible. And think about how many cloud customers they lost to AWS and Azure. TK came to GCP and his first work was adding an army of sales and account managers. This almost immediately yielded a rapid acceleration of the platform, although it's too late to catch up.
Scaling back up is really hard though. We had a de facto freeze on hiring (not exactly hiring freeze; more of a headcount cap) just shy of a decade ago to focus on our product. During that time, some of our best recruiters left because they basically had nothing to do anymore.
The freeze worked: we got rid of some products that weren't getting traction and were able to improve the products that did have traction. But the cost of the freeze lingered for at least a year; it reset the hiring pipeline, we couldn't grow fast when we needed to because the limited number of recruiters we had were already overworked, and the limited number of engineers had to balance interviewing needs with their real work. This all happened when my employer was <10% of its current size and pre-IPO, and we didn't even take a headcount reduction.
Twitter is simply at a different scale. 7500 -> 2500 employees is a 66% reduction. Going 2500 -> 7500 is a 200% increase. Recruiting is likely totally gutted, and the current 2500 employees have to support systems previously maintained by a 7500-person company. If they decide they need to grow, it'll have to restart at a snail's pace, and they'll have to make sacrifices on feature development or stability along the way.
Edit: for what it's worth, the fastest way to regrow back 200% is to rehire the people laid off. But, given that I happened to interview earlier today an ex-Twitter candidate who didn't make it through the Elon snap, that route is rapidly closing up.
It is not only hard, it also may or may not work. It is the same process Twitter has already went through years ago. I have simplified the issue and talked only about the product. I don't disagree with you.
This is not a trivial task. With such a heavy reduction and how entire teams have been completely decimated, there will be a lot of lost knowledge. I'm sure there will even be cases where the people who stay don't even know what knowledge was lost.
All that puts Twitter in a very risky position, specially in a product of such complexity that does a lot of things in-house. It shouldn't be underestimated.
> Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth.
I mean this is just wrong. Companies are always under pressure to cut costs (employees) and it is always talked about when quarterly results are posted. Look at how the market reacted to Facebook's latest results and then again what happened when they laid off thousands of staff.
The headcount at WhatsApp in 2013 was somewhere between 50-100, at which time they were servicing approx 400m MAU, which is more than users than Twitter has been able to boast for most of their existence.
Coincidentally, in 2013 SpaceX was just starting to provide commerical launch capacity, at which point I think they too had < 100 software engineers. A few short years later and they were re-using rockets, a feat many people had thought unlikely/impossible and requires some hardcore software eng.
Not surprised Elon Musk thinks he can run twitter with a skeleton crew.
1. WhatsApp was more reliable before migration to Meta's infrastructure.
2. WhatsApp didn't have e2ee back then. The broadcasting is important, yes, but it is very heavily biased towards reads over writes, so something like Cloudflare would solve 99% of the load.
My point 1 was specifically phrased to clarify how it could have been more reliable before migration to FB because it did not have to deal with the same load back then, you said nothing to show it was not a correlation.
Your point 2 sounds like there were additional factors that could have influence the reliability besides the load (they didn't simply migrate to FB infra but also switched to Signal).
> The broadcasting is important, yes, but it is very heavily biased towards reads over writes, so something like Cloudflare would solve 99% of the load.
There's push-notifying millions of devices within seconds after a celeb or a major news source tweets. There's tracking view and engagement stats on that in realtime. There's making sure a tweet is not available to any of those within seconds after it's been deleted or moderator. There're separate back-office apps for moderating that firehose of content. And that's just what I can see from the outside. An e2ee instant messenger with size-limited chat groups doesn't even come close.
Please don't say "just stick a CDN on top of it and you are 99% there", it's embarrassing (and not to twitter). This will maybe get you 80% there if your goal is "a microblogging platform" but not even 20% if your goal is being both the go-to news source and shitpost forum for people worldwide reliably working even in sensitive times and emergencies. Twitter used to be a microblogging platform back when it had much fewer employees and you'd see a fail whale regularly even as it had much fewer active users, in recent yeas it's a completely different beast and saying increased headcount is unrelated is amusing.
WhatsApp has scaled less than 10x since acquisition. They used to handle ~3M open TCP connections per server, and as a result could run their entire operation with under 300 servers.
The push notification argument is also overstated. Sharding and fan-out solves the burstiness. And people overall receive a similar number of messages (and thus push notifications) from WhatsApp as Twitter. Besides, these days the push notifications go through Google/Apple servers anyways to reduce the number of open connections needed on the phone side.
Then there are DMs. They are per person so CDNs don't help much (just static assets), but also they shard basically perfectly. So, shard them.
Which in the end leaves the user feeds. Designed correctly, sharding would work extremely well, and what doesn't work could be handled by caching closer to users for those 1k most popular accounts.
Honestly, with the correct architecture, languages and tooling, it could be handled by an experienced 50 person dev team plus another hundred in ops. Obviously Twitter doesn't have the perfect setup, so maybe an order of magnitude more? And if you throw a bunch of subpar engineers and tooling at the problem, nothing can dig you out of inefficiencies at this scale anyways.
And no, I'm not wildly optimistic here. StackOverflow still runs off of 9 on-prem servers [0]. I've seen message queues that can give 200M notifications per second on a single machine (written in C++, for HFT). This stuff is hard yes, but throwing more bodies at it doesn't help past the point your fundamentals are solved.
> WhatsApp has scaled less than 10x since acquisition. They used to handle ~3M open TCP connections per server, and as a result could run their entire operation with under 300 servers.
They switched to a new protocol and grew from 200 million to I guess about a billion users since 2013. If you believe a team of 50 developers could deal with this and not cause extensive downtime and service disruption along the way I pray you never ever manage software engineers.
> Sharding and fan-out solves the burstiness.
Great, at least it's no longer "just add CDN to solve 99%" here;)
> And people overall receive a similar number of messages (and thus push notifications) from WhatsApp as Twitter.
Yeah again WhatsApp has many users but as an engineer you just don't ever have to worry about delivering a message instantly to more than 32 people (512 as of this year), and you never have to account for moderating any of that because it's e2ee and there are no adverts next to the messages. It's basically dumb pipes terminated by one native client. Twitter has to maintain a mix of automated and human review of all UGC and is accessible via extensive APIs and search engine indexed web app in addition to native client.
> Then there are DMs
Let's ignore Twitter's DMs, even without them it's far more complex and demanding than an IM app.
> StackOverflow still runs off of 9 on-prem servers [0].
Yeah, and SO maintenance page or read-only mode is up about once a month and lasts dozens of minutes. What are you even talking about now bringing up a niche programmer-oriented help forum for comparison here?
You may be stuck in the times where Twitter was a RoR-based microblogging platform. It's not been that for years.
I do manage software engineers, focusing on HPC (image processing primarily), and one thing I consistently see from people who work with 'classic' web tech is underestimating what modern hardware can do.
This isn't 2005 anymore, we have multiple parallel 40gb LAN, 64 cores per socket and 2MB of L2(!!!) cache per core, and a full terabyte of RAM (!!!) per server. If you program with anything that makes cache-aware data structures and can avoid pointer chasing, your throughput will be astounding and latency will be sub-millisecond. How else do you think WhatsApp managed 300M clients connected per server without having just the in-flight messages overflowing memory, on top of all the TCP connection state?
Things only get slow when scripting languages, serialisation, network calls and neural networks get involved. (AKA "I don't care if you want docker, a function call is 10000x faster than getting a response over gRPC and putting that in the hot loop will increase our hardware requirements by 20x.")
The more distributed your architecture the more network overhead you introduce and the more machines you need. Running the WhatsApp way with less, higher performance servers simply scales better. Just from the hardware improvements since 2013 there was no reason for WhatsApp to change their architecture as they grew.
And if you think rolling out a new protocol while maintaining backwards compatibility is hard and somehow adding more people will help, I have a team of engineers from Accenture to sell you. I did this straight out of university, to thousands of remote devices, over 2G networks, with many of the devices being offline for months in between connections. You just need a solid architecture, competent people and (I can't stress this enough) excellent testing, both automated and manual. And the team that did this was 6 engineers, and this wasn't their only responsibility.
1) Their uptime was great. Regardless of what happened to their uptime after that (when they got to >1B MAU), they were already bigger than Twitter at that point, so...
2) Public broadcasts make a lot of things easier because that means more of your workload is relatively straightforward caching (as evidenced by this blog post).
Have you even thought about moderation and all the other concerns that go with this? How does instantly notifying millions of devices helped with caching, for example?
Now the systems are stable but human workers either be sick, leave, or die eventually.
Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them.
The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for the human workers unexpectedly not working anymore. Have a multiple workers doing the same things improve stability.
Sure, it's inefficient in terms of money. But alternative is one sick important employee catch a COVID-19 and die lost the knowledge of the system. Documents doesn't solve it because you want the manual operation available right now rather than a few months later when replaced workers learned from the documents.
> Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them.
People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…
>The only reason people work is literally for money…
Well, yes, but also that's super reductive.
The primary reason people work is literally for money. The specific _place_ people work is for things that interest them, engage them, challenge them, provide a path for continued growth in their career, or contribute to a common good they believe in.
And I would argue that it's not necessarily a given that people will be "more engaged and more excited about their work if they were paid more," but instead, may be willing to stick around in an unengaging, unexciting environment _longer_ when they're paid more.
That might be true for you, but way back when, I took a course in programming and thought it was the most fun I'd ever had. Couldn't believe that people wanted to pay me to do it. Spent the next 40 years making a great living. The only jobs I disliked was the ones where there were "glass bowl" bosses and co-workers. Problem solving was always fun. Not saying I didn't enjoy being paid well, but it wasn't the major consideration in whether I wanted to stay in any particular work environment. Here we're talking about the potential for huge schedule and technical pressure and low morale. A whole lot more money wouldn't keep me in those circumstances.
> The only reason people work is literally for money
Theories like Hertzberg exist for a reason and show that money isn't even close to the most important thing for people to work for. If the money is good enough, there are many other factors that are way more important. Not enough money is a reason for people to leave, getting more money is hardly ever a reason to work harder. The possibility of getting more money if they work harder is a good motivator though, once they have it that stops.
I've found that to be absolutely not the case, at least for me. If the job is super disorganized, boring, stresful for stupid reasons or just plain not interesting for me, no amount of money will make me engaged or excited. I've been in such jobs which paid a lot of money (for me) and the result was golden handcuffs - I didn't want to lose the money, so I was basically pretending to be way more engaged than I really was. I suspect majority of FAANG employees are like that - the money's just way too good to not put up a charade and keep milking the cow.
This really only works in situations where people are typically underpaid and need to get up to some living standard.
In most software engineering jobs, people have enough money to live an enjoyable life. At that point, having more money is not really such a strong motivator.
Yeah, and they were even profitable with ~3k employees. Then the hiring spree started and they went negative. Even if there wasn't Musk they would have to let go at least 30% of the people.
It's largely focussed on the event stream behind the core service and data analytics. There's maybe one entry on the main data store and one on search over the last few years.
Not just that. From my personal experience, twitter had to be one of the slowest websites I use. Even in my 2020 mac, it often shows the memory warning in safari. Things take a while to load. And the UX is terrible with having to constantly click to read child comments, having to click on “show hidden replies” etc. I honestly have no idea how a company with thousands of employees and a billion in loss was able to operate such a terribly performing website.
> What exactly where these thousands of employees actually doing
Maybe trying to increase traffic, attract advertisers, add efficiencies. Not everyone is an SRE. As long as your efforts increase revenue by more than you cost as an employee, you are adding value.
If the 3x headcount increase really did add no value, there are still about 1/3rd profitable employees there now. In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. The people that are the last to go are the ones who are very entrenched in the organization and who don't estimate their chances outside of it highly, and that's the exact description of who Elon thinks he's laying off.
I've found the opposite. I almost never see low performing employees fired outside of a mass layoff. In every layoffs I've seen 10x as many people were fired as quit. So you lose a bunch of low performers involuntarily, and a few top performers both voluntarily and involuntarily and that leads to the average quality improve.
My first job out of college had layoffs 8 months in. I was called into a conference room, connected to a speakerphone with the rest of my team, immediately told that we were all laid off and that I had some paperwork to sign in my managers office.
I was given a 33% raise, moved to a salaried position, and got a $5000 cash bonus 10 minutes later.
I had mastered the new system we were implementing and converted and supported the remainder of the customers from the legacy system. Shitty support of the product my team of now one actually drove revenue as the customers would pay higher support prices because of the shitty support I offered.
The service I offered was shitty because my job was to meet minimal contractual obligations for multi-decade contracts to persuade the customers to renegotiate the contracts for a product that hadn't been EOL'd. All the while offering onsite conversion services from the legacy product for most of my day job.
What I'm saying is that company is still alive today, they had a pivot point and literally cut 55% of their employees on a single day.
Google wants to beat facebook still, there is one of a billionity pivot points that twitter could take.
> In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking.
This is more true when the layoffs happen because the company’s situation deteriorates. If the company cuts jobs because revenues fall and products fail, better employees are indeed more likely to move to greener pastures before mediocre ones do. If, however, the company prospects improve, rather than worsen, this is no longer the case.
Twitter has lost a lot of revenue recently and is now in a lot of debt, although that might not have had time to sink in. In real terms it is in a much worse place, and remember that betting on a success is something you'd do as an investor, not as a rank and file employee.
I'm not sure. I think I'm a good employee, but I like job stability. If I'm working somewhere with layoffs, I'll likely go somewhere without.
There also isn't a measure of employee quality, and layoffs are often not about the individual (e.g. Amazon cutting robotics). I've had no clue how likely I was to be laid off during most points in my career.
This is the real question? Your question has nothing to do with the blog post and if you take a look around, what Twitter did was literally done across the entire industry, hence all the layoffs recently. There was a hiring glut to take advantage of cheap capital during COVID recovery. The capital has dried up, glut has ended and a lot of people lost their jobs. Why is that so hard to see? None of this is unique in any way to Twitter.
A bunch of people just got axed from Twitter because covid cash dried up? Shit I thought it was because Elon took over and fired anyone who refused to work at the office instead of at home.
The unemployment rate in Feb 2020 was 3.5%; by April 2020 it was up to 14%. It has never been lower than 3.5% since then. If there was a hiring glut due to covid it must have been a very industry-specific glut.
It definitely was industry specific. The need to satisfy a boom in remote working and people bored out of their minds in quarantine drove a ton of hiring in tech. If you had money to invest, tech stocks and companies were among the few places it made sense to pour money into during the pandemic.
There have been several front-page posts on HN in the past few weeks about big name company CEOs apologizing for massively over hiring during the pandemic not realizing that it wouldn't pay off once the lockdowns ended and people decided to go back to living their normal lives again.
What a wild card. Has Musk said why he purchased twitter in the first place? IRL he seems so introverted. It's just odd to me he'd have any interest in a social media company.
My pet theory is he got roped into it because the SEC was going to hit him for securities fraud (wouldn't have been the first time he'd done it[1]). Earlier this year, he had originally said he was buying a 5% share and would be a passive investor. Later on, whoops it turned out he'd bought over 9%[2] of the company without the mandatory disclosures, potentially defrauding the market of over 100 million that he would've paid in inflated share price if his moves had been known to other investors. With the SEC on his trail he said screw it, and inked a very difficult to escape contract to purchase the company. The SEC gets off his back, and he spends months trying to wriggle out of it but finally is compelled by a dead-end court case to fulfill his obligation.
He has a unique perspective as someone who's addicted to tweeting and whose account is constantly getting impersonation and crypto bots in the replies. His friends are incompetent VCs and other tech people who are convinced journalists and women with colored hair have taken over Twitter and are hoarding all the blue checks for themselves.
He's also extremely divorced, keeps having off the record children with his executives, and is having a midlife crisis.
Probably a combination of a career-first mentality where they don't have the time or inclination to seek / maintain lasting relationships, and also having the means to not raise their own children. To these people children are not burden in the same way as traditional parents because they hardly contribute to raising them; instead they are just another play-thing.
There may also be some game of thrones shit going on where if a boss and a mini-boss have a baby it solidifies rank and power.
Twitter was poorly managed. It would have been a $100B company in 2020-21 if it had shown any semblance of consistent profitability.
The opportunity here is to:
- Take the company private
- Cut costs by pruning staff and reducing bloat
- Get high-engagement accounts back, which, in turn, will bring back advertisers (perhaps sweeten the deal by promising something with your other ventures)
- Eke out a few quarters of consistent profits
- Go public again, this time with $500M/quarter of profits
- Enjoy the standard "Musk-effect" 40x PE multiple and slow keep cashing out
Hard to cut costs when you've just saddled the company with billions of dollars in new debt. Don't forget El Musk-o is getting sued by Tesla shareholders over his bloated compensation.
Get high-engagement accounts back
Advertisers didn't leave en masse because "high-engagement accounts" left, they left because the "high-engagement accounts" came back. From the looks of it "absolute" free speech doesn't include Alex Jones, so who knows what other "high engagement" accounts will be left out in the cold. Doubling down on "absolute" free speech just means you're going to have a sea of ads about pillows, dick pills, and reverse mortgages.
Can you trust anything Musk says? We know he’s full of shit; ask Thiel. As for the real reason, pick your poison:
- He purchased it because he’s in a right-wing filter bubble and bought into the idea that Twitter is controlled by a secret left-wing cabal and genuinely believes he’s doing the world a favor because he thinks right wing propaganda is actually moderate
- He has the same brain damage as Trump and requires adulation by sycophants and he thinks this is the best way to guarantee his fix and give him control over his critics
- He wanted the brand name and doesn’t care about what services are currently offered under it
- He’s from the southern hemisphere and wants to destroy the social fabric of the United States
- Some permutation of all of the above
Not sure which of these are the truth, but I’m guessing a combination of the first two.
Musk is an excellent strategist. When he says something in public it's because he wants people to react to what he said. It doesn't mean it's true. I'm not suggesting he 'lies' exactly, more than he uses his audience to manipulate the world around him to get him closer to his goals. His Twitter account is more PR than reality.
I have no doubt he had excellent reasons to buy Twitter, but what he says they are in public probably isn't the whole story.
Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc.
How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!"
These qualified people thought at the time it was a good idea to get up to 7.5k people. How were they all wrong?
It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team.
Organizations tend to bloat.
Random, rapid cuts might not be the fix here, but headcount was too high.
Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up".
Or I tell my boss "code tends to take longer than estimated to deliver".
I wouldn't be given lots of promotions/bonuses with that outlook.
We're talking about a Twitter CEO with $30m/yr+ in total compensation.
There's no way his/her view was "shrug, organizations tend to bloat" while racking in extremely competitive + good pay as somebody whose main job is to drive a company towards maximum growth/profitability.
There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.
We, the people on the outside looking in, have to be missing something.
> There's no way his/her view was "shrug, organizations tend to bloat" while racking in extremely competitive + good pay as somebody whose main job is to drive a company towards maximum growth/profitability.
> There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.
Why not? Why would those board members have ever called him out? The more they're paying him, the more they can pay themselves too.
Yep, my understanding is that cronyism between the executive and the board is a huge issue driving CEO pay. (Boards can also just be out to lunch.)
One way to fix this is to make corporate takeovers easier. This sort of cronyism ends up being a con on the shareholders (and the con is especially easy to pull off if the "shareholders" are passive index funds that don't pay much attention to their holdings).
By removing legal protection on corporate raiding, the board+CEO have to worry about activist investors who ask inconvenient questions like "why are you paying yourself $30M a year instead of giving shareholders a dividend?"
This is a appeal to imperfect authorities that make mistakes all the time. By that logic, Elon was able to convince banks to loan 14B to acquire and gut Twitter. There was no way the staff and shareholders of those organizations, much larger and more proven than Twitter could have been so complacent and ignorant to make such a mistake.
Elon himself is a CEO with a better track record than Parag, why is he not more legitimate if we are pretending leaders can't make mistakes
This is exactly what happens. It's also why CEOs buy other firms even though research shows the majority of takeovers fail. Just pick up any management book in the last 20 years.
There's a clear correlation between org size and CEO pay, even when not in the company's interests. Some CEOs and managers also love the power. Politics is also at work.
> Or I tell my boss "code tends to take longer than estimated to deliver".
I've used variations of that phrase throughout my career. I'm very well paid, partly because I account for this and make certain that other stakeholders are aware.
> We're talking about a Twitter CEO with $30m/yr+ in total compensation.
Jack, who was once the CEO of Twitter, has said that they hired too many people too fast.
Elon, who is "paid" way more than $30 million/year to run several companies, also apparently thinks it was very bloated.
I'm not necessarily saying the billionaires are right and the millions are wrong. I'm just pointing out the clear counterexamples to your appeal to authority. I honestly don't understand why you would be so surprised to find out that millionaires also make plenty of mistakes. The Twitter stock price to a degree is one indicator that the Twitter executives were making mistakes.
> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!"
The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I shouldn't. Maybe the effect of all that extra money sloshing around in the system was to incentivize hiring everyone to make sure you didn't accidentally get a false negative, and not all of those hires were good ones.
A friend of mine used to work for a private equity firm that had about a hundred C++ and Java developers working on a stock trading and prediction system. He quit and rewrote the essence of that system in F# single-handedly.
If you watch interviews with famous developers like John Carmack, they'll mention that working alone scales to about the equivalent of 5x developers. That is, adding 1-3 extra people might slow you down because of the overheads of communication and coordination. It's only around 5+ in a team that there is a definite advantage.
But what are the chances of putting together a team of 5 rockstar developers that all agree on language, style, and vision? Basically zero. So you have to settle for mediocrity. Popular languages, simple approaches, established design patterns.
If you're an experienced "rockstar" developer coding by yourself and use a fancy language like F#, you can outperform a team of 10+ people. If you're replicating a system you've seen already, 20-50 might not be out of the question, especially if you're smart enough to avoid "tarpits" and instead rely on good quality libraries and CotS components like databases, PaaS, and the like.
I think this is not too far wrong. I also think in addition that the really competitive job market of the last 2 years and the hiring they were doing inevitably resulted in some highly paid people who were not good fits or good at their jobs. I think that covid wage hikes and job competitiveness is doing a bit of reckoning now as there is finally enough data to evaluate performance relative to output.
OTOH, at non tech companies, all the research interviews and surveys I have done recently continue to show a talent shortage and fears of losing tech staff, so maybe it is just that Tech companies are saturated and have too much capital for their creativity.
I'm not doing interviews or surveys, but this also matches with my anecdotal experience with non-"tech" companies, who are all still starving for talent. I guess it makes sense, given that tech companies disproportionately pay in stock and tech companies were disproportionately highly-valued during the 2020-22 Covid weirdness.
>OTOH, at non tech companies, all the research interviews and surveys I have done recently continue to show a talent shortage and fears of losing tech staff
I wonder if any science fiction writer could have predicted silliness like Snapchat having fewer outages than serious government websites because Snapchat is better at hiring engineers. Tech hiring is so bizarre.
The problem with having a lot of money sloshing around is that you can no longer say “no”. You estimate a new feature will take your team a year. But someone wants it in 6 months. “Why aren’t you hiring to meet the deadline?”
My first job out of Uni was a company that ran financial services, logistic fulfilment, and retail. They had at least a dozen large in-house retail brands, several large warehouses with automation that Amazon was probably behind at the time, and they also ran serviced some of the most well known brands in the country - from their website to customer support to logistics. They had delivery networks, networks of direct shipping, vast call centres with thousands of staff. We integrated with the banking system, with all sorts of external companies.
Our total headcount was under 10 000. Most of that was logistics and call centres (taking retail orders). Our IT dept was literally about 200 ish. All our office staff fit within a few floors of a medium sized building. IT was less than a single floor.
I’d say our software, whilst serving fewer users, was completing tasks orders of magnitude more complex than Twitter. We didn’t have much down time, in fact almost zero beyond physical damage to our networks and the once every few years disaster. Our critical support team was a handful of greybeards who spent much of the time playing solitaire. Code reviews were tight and access to production strictly controlled.
What we did have was a very lean culture, carefully managed over a long time. The company was over 100 years old.
I do not understand how, with far more sophisticated tech, companies like Twitter need so many staff. We could have run Twitter with a thousand folks, including sales. From an engineering standpoint a few hundred would do. And most of us weren’t great engineers. There were a few very experienced freelancers, though. We all worked 0900-1730, but we worked 7 full hours every single day, every minute of those 7 hours. No beanbags. Most of that time was sat quietly coding in a very cleverly designed open plan office (no cubicles) that was like a library.
Looking back, I think the key insight was that we were very cost driven and absolutely kept everything as simple as possible. We had all the usual project overrun problems, usually because things turned out to be more complex than previously imagined. But it baffles me when I hear engineering head count at places like Twitter. If you had given us 1000 engineers, we wouldn’t have known what to do with them.
It is amazing how quickly complexity comes about to serve complexity. Like at $CURRENT_JOB we split everything into a few dozen microservices with their own DBs. Turns out that we need all the data in one place to make some decisions. Shocking I know.
Cue a team of engineers working for a year+ to build a complicated data pipeline to bring all that data back together into a graph DB (not a clue why a graph DB) and build a DSL to write the code to make these decisions.
That's where the engineers go. It's complexity that is introduced setting off a chain reaction of increased complexity to account for the complexity introduced. Then you need complexity to account for that complexity and so on.
Not that it's all self inflicted wounds. Running a site that counts 1/10 of humanity as active users is a hard thing to do
> These qualified people thought at the time it was a good idea to get up to 7.5k people. How were they all wrong?
Come on, just look at the tech industry. When rates were low and stock prices kept going up, "headcount" was used as an indicator of future growth. Grow headcount, investors are happy. After all, the promise of tech stocks was "growth". Usually you're not looking to cut costs until you think growth is over. Of course, Twitter was a dog and did nothing useful for years, no innovation, no new products, nothing. But tech investors definitely saw rising headcount as a good thing...
Indeed. I watched a 30 headcount company that was quite productive getting ballooned up to 170 people after an acquisition. You won’t believe it, but productivity dropped _sharply_. Processes have been introduced that emphasized safety and risk mitigation so heavily that whole product lines are scratched before they can be released. Judging by products and improvements they released within the last year, you could be tempted to think they just stopped working altogether.
And the leads installed by the acquiring company seem to be highly intelligent - the company continues helicoptering in their brightest minds to fix this mess. It just so happens they are forced out every 12-24 months because it’s almost impossible to deliver anything there.
For example, last year they introduced SAFe. That caused an exodus which basically equaled a brain drain (still ongoing). Most people I valued for their straightforwardness and ability to cut through BS there have left or are leaving. The person pushing for SAFe introduction has resigned already.
The founders of that company have all left ofc, and are basically trying to restart the same company (or variations) with their own money. And their products are already picking up speed, becoming threats to the original one.
>Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"?
For the same reason that colleges and universities have seen their administrative bloat skyrocket at 10x the rate of student enrollment. Administrative bloat inevitably creeps into all large organizations. Many of the people in the trenches making hiring decisions weren't considering the overall financial performance of Twitter as a company. They were making hiring decisions based on what was happening in their own department, or how that decision would help advance their own agenda, or increase their budget, or increase manpower on a favored project. When you further consider that many at Twitter openly conceded (and in many cases, bragged about) that they viewed their role at Twitter as moral arbiters of society, crucial to policing the discourse of the public, it is not hard to see how enlisting as many true believers as possible to the cause would be seen as desirable, regardless of the larger financial implications.
> These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc.
Wealth is not a valid indicator of ability.
I'm not judging the execs and board members individually but rather questioning your assumption. I have read you mention "supposedly", yet it can be read as a rhetorical term.
I've worked in multiple financial services companies where management is incentivised to be as ruthless as possible and they are always overstaffed in areas and understaffed in others. I've been in teams of 10 people that could be staffed by 2.
Hiring often isn't done because of current requirements. Senior execs come and go and with them so do strategic objectives. You accumulate people and they're often not laid off when the thing they work on becomes redundant. Large scale layoffs are awful for morale and usually only come after a 'crisis' occurs.
Has no company ever been mismanaged? Have they they ever grossly misallocated funds? The answer is of course yes, that happens all the time. Corporate leaderships are not not infallible.
All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics.
Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart.
The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or elicits help from others on what to say that sounds like tech speak (RPCs!), but when it comes to being truly knowledgeable in this space, he is losing his most valuable assets because of his amazingly poor managerial and ownership style.
I know there are a lot of Elon fans on this site, and will disagree with all of this; but his abilities have not at all been proven. Yes, he knows how to spend money to claim credit for technical advances, but until he actually has his hands dirty in the muck of the hard work of tech, he will always be a glorified self-promoter with no substance.
John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so."
Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction.
He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy.
He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years."
Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it works about as quickly as you'd expect a PhD student to. The fact that he can do this across many technical disciplines is impressive and borderline superhuman. I don't think people understand or would believe how low-level and technical typical meetings with him are. Just saying because I get triggered reading way off innacurate takes on this topic (original comment).
I think what upsets a lot of the Silicon Valley types here on HN is that people just like them are being called on their bullshit and fired en-masse for it. That has got to be uncomfortable.
You know the old saying: "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It."
That's what's happening here.
That engineer that was humiliated publicly for defending a slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine? Hacker News is filled with people just like him that have built near identical software in other orgs.
Understandably they're upset and are looking for any excuse to dismiss Elon's criticism of not just Twitter, but their entire industry.
I doubt you'll find many people arguing that the value of employee output is evenly distributed across all people in a company: regardless of discipline, there's always a minority who are delivering the most value. There's always the most valuable software engineers, the most valuable sales people, the most valuable executives. The problem with Elon's specific brand of this take is that it's ignorant of the real-world human aspect: Elon could pick the best software engineer he has ever worked with, and helicopter them into a dysfunctional environment, and they would struggle to deliver value.
If Elon had joined Twitter, and spent time understanding the business and environment and then excised the people he felt weren't contributing towards his vision, that would be one thing... but he has made arbitrary judgements based on absurd metrics like lines of code or willingness to show up at 1am to draw on a whiteboard, he has not made judgements based on the quality of the work or the value people have delivered.
Likewise, to suggest that a software engineer is bad because they were a part of a team that built a "...slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine..." is absurd: what if that person was the only reason that it wasn't 10x slower? What if, they were the lynchpin in that team ensuring that brought everyone else up to a much higher standard which ensured that what they built was usable (even if it was bad)? You cannot judge the contribution of an individual without considering the wider context.
I have no problem with a company cutting most of their software engineers (I encourage clients to minimise their exposure to software engineers, I encourage careful hiring over volume) but what Elon is doing is... not that.
> ...spent time understanding the business and environment...
You're probably right, but there's a decent chance that Elon's heavy-handed approach was necessary.
I've seen the "gentle" approach fail.
For example, at $dayjob a bunch of on-prem stuff is being slowly modernised into the cloud. Very slowly. Slow enough to give the dinosaurs time to play politics and protect their turf.
For example, the networks teams that are used to legacy in-line firewalls will cozy up to some non-technical senior manager with a budget and get them to approve a project to roll out this legacy technology in the cloud. That way they don't have to retrain or -- worse -- risk being made redundant.
If instead some team comes in and simply bulk-migrates workloads from on-prem to the cloud... breaking a handful in the process and just fixing forward, then it appears to be messy and crazy, but the effect is that the legacy data centre teams are made redundant virtually overnight. Now they've got no clout, no time, and no pull. They're simply walked, and will find jobs elsewhere.
I've seen both approaches, and the latter style worked better long-term.
Every org has slow-as-molasses, badly designed, illogical components. It is about ratio.
Time will tell. I don't think Musk/twitter's case will set any precedent. He is too much of a character to provide broad meaningful insights into industry. Also he has accumulated a list of failures which are rarely mentioned.
If that is the case, why has he been making mistakes that seem fairly elementary on Twitter? Like, I understand not understanding a problem space and wanting to learn more. But you say he has good intuitions and mental models–I would've expected at least some basic background research before posting online. Why aren't we seeing that?
Has he ever make a public statement, at least a paragraph in length, explaining something technical?
It's hard to blame people based on his decades of public behavior and lying about his education, falsely claiming to have a physics degree and to have been admitted to grad school.
> "Anyone who actually writes software, please report to the 10th floor at 2 pm today. Before doing so, please email a bullet point summary of what your code commands have achieved in the past ~6 months, along with up to 10 screenshots of the most salient lines of code"
Actual quote.
Anyone using the term "code commands" comes out a little detached from programming reality, let alone the rest of this request, it is out of a Dilbert strip.
"Code commands" is very plausibly an autocomplete flub of what was supposed to be "code commits." When I type "code comm" my iPhone offers up "commands" as the completion.
I've seen a lot of mockery of this request, but I suspect people aren't considering the wide variance in employee quality that can exist within a mismanaged organization. What Musk was asking for here wouldn't be a good way to evaluate skilled, conscientious developers, but it would be a pretty effective way to rapidly identify people who are basically incompetent or just aren't really doing anything.
> What Musk was asking for here wouldn't be a good way to evaluate skilled, conscientious developers, but it would be a pretty effective way to rapidly identify people who are basically incompetent or just aren't really doing anything.
So it's basically a FizzBuzz test, but for existing employees?
Many people who have worked with Musk have shared similar sentiments in interviews. But it seems that people just refuse to believe any of it. People think that there's no way it's possible for someone to be that deeply technical and be a CEO of multiple companies at the same time. I've talked to people about it and they straight up refuse to believe it saying that it's impossible and that any evidence of him being technical in interviews is all set up and that he was trained on the materials and questions ahead of time.
With a handful of tricks or a patsy in your pocket, it's easy enough to pull things like this off. These are all self-reported encounters, which lends some doubt to them; as I've never seen any public performance of his that suggests he has this exceptional intelligence or that he isn't subject to the same amount of irrational thinking that most humans are. You may be able to do some type of rocket equation in your head, but if you constantly promise things that aren't ultimately delivered.. people have good reason to question this narrative.
He clearly does know how to make incredible sums of money. Why that's not enough and people need to find excuses to exaggerate or demean his intelligence is beyond me.
> but if you constantly promise things that aren't ultimately delivered.. people have good reason to question this narrative.
This is the insane thing to me. He's promised a lot of things, but he has also delivered some pretty huge things. Tesla kicked off the electric car migration and has millions of EVs on the road. SpaceX has reusable first stages on their rockets and are the only private company to send humans to space. Just those two things alone are massive achievements. But people look at some things he's promised but has not yet delivered and that somehow is more important than what he has delivered?
Maybe I'm a particularly dull engineer, but I've taken several aspects of personal advice from what he has said in interviews (the especially technical ones, not the ones aimed at a mass audience where he repeats his standard canned speech) and found them useful for myself personally.
Here's two examples I've found particularly insightful that shows he has some ability to talk about engineering details.
This example where he talks about the choice of steel for Starship as opposed to any other metal, something that would be an otherwise unsual choice: https://youtu.be/vLC5W53Fsyg?t=936
This example that I've personally incorporated into my own thinking where he talks about his "five step process" for engineering design refinement (watch at least until he starts talking about Tesla Model 3 battery stuff): https://youtu.be/t705r8ICkRw?t=805
>This example that I've personally incorporated into my own thinking where he talks about his "five step process" for engineering design refinement (watch at least until he starts talking about Tesla Model 3 battery stuff): https://youtu.be/t705r8ICkRw?t=805
I knew what you were talking about when I started reading your comment. "Make your requirements less dumb" first seems so obvious once you've learned it.
All the denigration directed at him seems to come from people who've only read headlines about him from sources who hate him.
Something doesn't compute in this scenario though. Either his tricking everyone around him or is unfortunate enough to slip up publically. Not knowing what GraphQL is and talking about RPCs in HTTP is a very revealing slip up.
My guess would be that he has some knowledge but also is very good at faking it which is not necessarily a bad thing - those are good traits for a CEO. Though people should be aware of this fact when evaluating the whole persona.
I'm inclined to believe this is true. The problem is that Twitter's challenges are social/political, not technical, and Musk has demonstrated little competence in this area.
This is the absolute root of what's going on, right here. Twitter is only nominally a tech company; it's a media company. It may be that he had to cut the fat over there, I don't take issue with that necessarily (though I certainly do take issue with the disrespectful way he went about it), but image is incredibly important at a media company and he's notoriously bad at comms except with a small subset of people. Twitter needs sensible policy and thoughtful communication, and he wants to ram his ideology through it like he would shake up any technical process.
Another area where Musk's "brilliance" has faltered is in his transportation ideas and there's a connection here to twitter in that the challenges here again are not technical, but rather political around land use.
Brute forcing a problem with better technology not always the actual solution when technical problems aren't actually the problem.
Channing Robertson, the face of Stanford chemical engineering department and the associate dean of Stanford’s School of Engineering, who taught and mentored Elizabeth Holmes, has said the following to say about her:
“She had somehow been able to take and synthesize these pieces of science and engineering and technology in ways that I had never thought of.”
“I never encountered a student like this before of the then thousands of students that I had talked”
“You start to realize you are looking in the eyes of another Bill Gates, or Steve Jobs.”
He also maintained that Holmes was a once-in-a-generation genius, comparing her to Newton, Einstein, Mozart, and Leonardo da Vinci.
Excerpt from: "Bad Blood: Secrets and Lies in a Silicon Valley Startup" by John Carreyrou.
Adding another datapoint from one of my previous comments:
In response to someone saying on Twitter how Elon doesn't understand the technical stuff of rocketry, Tom Meuller, former CTO of Propulsion at SpaceX and the designer of many of their engines responded
"I worked for Elon directly for 18 1/2 years, and I can assure you, you are wrong"
You know, I think Musk is an ass, and would never work for him, but don't you think that someone who has managed to launch and then run many successful and complex technology projects might actually know a thing or two about launching and running simpler technology projects?
And if you're going to claim that his successes have been due to the people surrounding him who actually know what they are doing, then all that tells me is that you are acknowledging that he knows how to surround himself with people who know what they are doing.
We're not fans (I'm certainly not), but it takes a special kind of mind to look at Musk's track record of successes and conclude that his latest project is doomed.
Well, I think the issue is precisely considering Twitter a "simple technology project", and it's the same mistake that Musk does. Twitter isn't a "software and servers business" as he said. Twitter is a social community, and while in some regards it might be easier, it's also far more difficult in others. Just compare how many business and institutions can reliably launch rockets or create cars, and how many can reliably create social networks.
I think it's somewhat reductionist to call Twitter "simpler". The technical challenges faced by SpaceX, for example, are almost completely orthogonal to those faced by Twitter. Imagine swapping a random engineer at SpaceX with a random software engineer at Twitter -- do you think either would thrive in their new role?
On the contrary, I'm pointing out that "simpler" is meaningless in this context. The two companies face fundamentally different challenges and require completely separate skill sets. Some people will be better suited for Twitter, whereas some people will be better suited for SpaceX. For the same reason that you shouldn't commission a mathematician to remodel your bathroom, you shouldn't expect the CEO of SpaceX to successfully manage a social media company.
For further reading, I'd recommend this article in the Atlantic. The relevant portion is about midway through.
Compare how many companies can reliably launch rockets vs how many can reliably create social networks that aren’t dead in a year. I think it’s clear what’s more difficult
Point still stands if you call it a "simpler technology project". It's not about the tech, it's about the community, and communities are far harder to predict and manage than a rocket or a car.
> Point still stands if you call it a "simpler technology project".
Which point?
> It's not about the tech, it's about the community, and communities are far harder to predict and manage than a rocket or a car.
While I broadly agree with you, and also agree that twitters founders and/or leaders are better equipped to manage a community than Musk is, I fail to see how having someone with a track record of success (some in turning around failing businesses) automatically dooms the company.
That was the point I was responding to, in the original post. It's also why I ended of with "it takes a special kind of mind to automatically assume something is doomed just because someone with a track record of successes took it over".
I don't think Twitter is automatically doomed, but I think there are more things to consider than just Musk's "track record of successes". And one of the important things is that he doesn't look like he actually understands Twitter. He keeps calling it a "software and servers company", keeps talking about "hardcore coding", alienating and angering users and advertisers... People would be be more confident in his abilities if he actually looked humble enough to recognize the parts where his expertise is lacking.
America is such a great country that a random person can just fecklessly blunder into creating a revolutionary electric car company and cluelessly blunder into creating a rocket company that is the envy of the world.
Automotive and aerospace are not that similar to social media. People buying into the vision of "get the planet off fossil fuels for transport" and "get this species to Mars" are probably willing to make sacrifices that people working on social media are not.
It's the Halo Effect fallacy to think competence in one field automatically translates to another. Especially when the founder in question has displayed increasingly erratic behavior in the meantime.
Is today's Elon capable of doing what Elon from 15 years ago did at Tesla? I don't think that is necessarily in evidence, much less in a very different industry.
There's a lot to unpack here, for one do we consider those two enterprises to be successes? It seems too early to tell - they do appear to be influential, but the C64 and Palm Pilot were influential without being successful. It's also not clear if they are long term successful (I think Tesla will be, but SpaceX is very much currently dependent on government funding). Finally, it's not clear whether Musk was a critical driver of success - evaluating his contributions based on the outcome of a company is basically resulting.
Look, I don't know if Elon is a genius or an opportunistic parasite with really good PR. It seems unlikely if we ever will know that. What I object to is people pointing at his ultimate financial success and crediting him with the current result of 2 big companies whose future is very much not determined.
When I look at his process from this ant's perspective, I think he is an abusive unstable individual who takes credit for everyone's work and lies a lot. He also flip-flops depending on the wind. Is that success? Not based on my personal values. Have his companies accomplished a lot? Some of them, absolutely.
The definitions and evidence matter a lot, and I personally don't think any of us are qualified to make blanket statements based on incomplete outcomes. Further, I don't think his other companies that require primarily good engineering are very relevant to inherently people problems, like Twitter. My evaluation of how Musk handles people problems is that he is very bad at them, and I anchor my prediction about his Twitter leadership based on that.
What I think makes people skeptical of him is that deep down we all believe in nominative determinism. Who do you think runs a rocket company behind the scenes, "Shotwell" or "Musk"?
> Automotive and aerospace are not that similar to social media.
Yes. Social media is easier.
> It's the Halo Effect fallacy to think competence in one field automatically translates to another.
This is precisely about leveraging the Halo Effect fallacy. Elon Musk might not know social media, but the markets don't know that, nor do they care. The average retail trader sees "Elon Musk's company" and buys and holds, regardless of absurd PEs.
Musk knows the power his brand has. He's simply going to use that to pump up Twitter's valuation, all through the virtue of his "halo"
Dealing with people is generally far harder than pretty much any engineering problem. The same is true of Twitter, because there are no easy answers or even clear goals.
Exactly! I can't fathom that people don't seem to understand there's a comparable amount of new companies started every year in the space, automotive, and social network categories.
The easiest way to do this would be to make the site completely free of moderation, but that quickly becomes a cesspool. Are there other ways to define the core audience?
This is just a misdirection. I could just as easily said, nobody did it by themselves, including Musk.
What is “it” in this case?
NASA (and the DoD) had vertically landing reusable rockets designed for orbital flights back in the early 1990s. They were being successfully tested but budget cuts killed the program. They weren’t doing “it” because it wasn’t the same priority in that era. NASA has been researching COPVs for decades, etc.
The SpaceAct agreement between NASA and SpaceX allows for sharing of this kind of information. If you think SpaceX has done all their great work alone, you are misinformed and likely making you data fit your conclusion instead of the other way around.
SpaceX has some competitive advantages, but I don’t think they are what you think they are.
> If you think SpaceX has done all their great work alone
What I'm saying is SpaceX got it done. No other organization in the world did. The fact that SpaceX had an obvious learning curve of failing and exploding rockets makes it obvious it wasn't just copy and install NASA technology.
NASA's reusable rockets were on the space shuttle, which turned out to be fantastically expensive and impractical.
> NASA has been researching COPVs for decades
Somehow not resulting in a practical, inexpensive reusable rocket.
It's undeniable that NASA has made many great achievements. But making space accessible in an economic manner isn't one of them.
>NASA's reusable rockets were on the space shuttle
I wasn't referring to the shuttle. Note I said "vertically landing reuseable rockets".
Are you claiming SpaceX doesn't use COPV's, or that don't benefit from prior COPV research? I don't think either position is accurate. Of course no single technological advancement defines space exploration.
Can you elaborate on what you think SpaceX's key advantages are? I can use that to gauge if you really know what you're talking about. I don't want to sound rude, but it's starting to come across as a poorly informed discussion, but one where you have strongly held beliefs. That's not the relationship we should probably hope for.
I already said the NASA/DoD program was scrapped in the 90s. They had successful sub-orbital test flights, the original designs were for an orbital craft, but the project was canceled before that could be tested.
I meant what advantages do they have to facilitate that. You gave me the outcome but haven’t shown any understanding of the why. When somebody asks what makes Tom Brady special, saying “because he wins more” isn’t really saying much and doesn’t take show you know much about football.
I ask because I suspect you will just give some rote public vs. private answer but that’s only a superficial reason. There are underlying systemic reasons, but you need to remove yourself of that false dichotomy first to get there.
FWIW, I’m not a big fan of NASA. I think they are largely a broken culture and a shell of what they were in the 1960s.
This is exactly the kind of vague, hand-wavy answer that I was hoping wouldn't be the response. It doesn't really show any understanding of the problem. The issue with the "profit motive" as an explanation is that it's so vague it can be used to argue both sides at the same time. The profit motive helps them pay for the best-and-brightest. It also biases them to hiring the least experienced at a cut rate. It incentivizes them to provide the best product. It also incentivizes them to cut corners to save money. It explains why SpaceX has been able to move fast; it also explains why moving fast caused such problems with Boeing's Starliner. So the "profit motive" doesn't really explain anything. Besides, the vast majority of NASA work is done by for-profit contractors and has been since the Apollo era. There's nothing new about it in spaceflight.
I'll try to illustrate a more nuanced perspective. It's no secret that NASA levies a lot of tough requirements. For example, contractors must have a robust pressure systems program. This includes managing/certifying systems all the way down do small air compressors in a vehicle maintenance shop. Same goes for software quality and a million other aspects of spaceflight. Contractors hate these types of requirements because they're expensive. Many within NASA hate them, too. Certainly some of this is bureaucratic overreach, but a lot of it is also good, sound engineering practice. There are mechanisms to waive these requirements, but few people want to openly do so for a variety of reasons. I could go deeper into the why but it's a bit of a digression.
So what does this have to do with SpaceX? CCP, IMO, is a clever work-around to avoid accountability to these requirements. NASA, rather than buying a product, is buying a service. So even though NASA expects them to meet those same requirements, there's very little oversight to force them to do so. Some people have raised flags about these issues but are essentially told to stand down because they don't want to tell the contractors how to provide the service. It also gives NASA a smokescreen to get what they want (faster, cheaper production) while avoiding accountability when things go wrong (they can always point to the requirements they claimed they wanted, but didn't provide the oversight to ensure). NASA knows those requirements rapidly increase costs and on one hand they don't want them, but on the other they want plausible deniability if something goes wrong. Minimizing requirements can streamline the process. The fact that they manage way less requirements is why SpaceX can have a single 23 year-old managing a program that takes a team at NASA. In other respects, it turns a blind eye to the very requirements that manage risk.
E.g., Falcon 9 had supplier quality issues that lost a rocket [1]. Most who work with flight hardware would be surprised to learn SpaceX wasn't applying industry-standard supplier quality checks on critical flight material. Once a mishap happens, NASA gets to swoop in and investigate. And the result is SpaceX now has multiple reliability layers to mitigate that risk. It's not that it was some unknown risk, it's just that they weren't managing it properly. To the uninitiated it looks like they were running a tight, streamlined ship but in practice it was being played a little too fast and loose. Boeing did the same with Starliner but had a bad roll of the dice. The real question is how many times can this happen before SpaceX starts to look like their bloated competitors? NASA could do the same by just peeling back requirements and upping the risk.
Starliner also had it's own host of quality issues as part of CCP. People at NASA were concerned, but their hands were essentially tied until there was a smoking gun in the form of a botched demo that risked crashing into ISS. Again, the "profit motive" at work can sometimes mean more risk than intended.
As a different example of risk, SpaceX can create an assembly line of manufacturing to reduce financial risk. NASA can't because they they are also forced to reduce political risk by spreading programs all around the country. It's not that NASA is so incompetent that they can't figure it out, it's that they are managing a different set of risks. NASA manages the risk that ensures funding for SpaceX while SpaceX manages the financial risk of manufacturing. It's a symbiotic relationship, but one that can be misconstrued by the ignorant as some public/private dichotomy.
Yet another example of risk: Lots of people want to point to SpaceX rapid iteration as a strong point. It is, but people also need to understand that rapid iteration is also at odds with reliability. SpaceX may just quickly change a design (see their COPV mishap) but it's up to NASA to figure out the true failure mechanisms on their own dime. This rapid iteration is also why Tesla's quality measures are usually quite bad. They can't stabilize a design or supply chain long enough to generate high quality.
Suffice to say, there's lots of good and bad tradeoffs of the public/private dynamic, but just saying "it's the profit motive" explains none of them.
Judging by some of the old patents he's filed [1], I'd guess he has at least a decent understanding of the tech involved. Probably less so, when it comes to the details of more modern distributed systems, but I also wouldn't be surprised if he's spent some effort towards all that as well - he's been working in/around pretty cutting edge tech for quite a while. Could he sit down and code it himself? probably not, but that's hardly required in his situation.
You aren't wrong, but his playbook is familiar to anyone who's gone through acquisitions (especially leveraged ones) and many companies were in a strong enough position to start with that they do manage to limp through and get sold off despite all the abuse.
You underestimate Elon Musk. Many people have done that before and lost that bet. If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies. The notion that he won't be able to attract world class talent is ludicrous. Yes, he is a bit of a liability and his management style is obnoxious and unconventional. But he does get things right once in a while.
And he hates bloated inefficient teams. His decrees on meetings are infamous. Tripling the team at Twitter implies a lot of internal politics, fiefdoms, communication overhead, and generally a lot of headless chickens running around. There's no nice way to fix such a team. A sledge hammer is one way to fix it and obviously he likes getting results quickly.
So, the notion of laying off most of that team was a foregone conclusion. The notion that a lot of the better people would get upset about that and leave as well is also highly predictable. What's left is a team with some gaps but also a lot of breathing room. And he can always lure key people back in by throwing money at them.
Simple plan. It might actually work. At the cost of a bit of drama, temporary instability, and lots of free publicity. Exactly his style. Cringe worthy and effective. I can see the logic here.
Agree, I wonder how much it is thought through strategy and how much is just "natural" style applied indiscriminately. I think one more important part is that he has money/resources to be able to make mistakes without bankrupting and stubbornness to plough thru even when things go wrong.
> If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies
The point is that Twitter doesn't really needed someone to build a world class software and hardware team. The technical challenges in reliability and speed seemed pretty much solved or on track to be solved already. The problem of Twitter was that they never knew how to properly manage the community and make the company profitable.
Twitter doesn't have a tech problem, it has a community problem.
Exactly. it had a team problem. I think it's safe to use past tense now because that team is mostly gone now. It still has some team challenges but those he can fix with strategic hires and hard work.
Fixing the community starts with rolling back all the things that clearly did not work. He's using the sledge hammer method there too. So, not very subtle but generally just getting of rid of a lot of failed and failing policy.
The technical challenges in speed and scaling are not challenges at all anymore. Twitter built a lot of stuff in house when you couldn't get that stuff as a commodity. That has changed since then. You need a cache, you can get one from any number of cloud providers or spin up something off the shelf you run yourself. Same with databases, CDNs, large scale object storage, search infrastructure, message brokers, and all the rest. So, yes, there might be a need for changing some of that necessitated by some key people disappearing but it's not a massive technical challenge.
> Fixing the community starts with rolling back all the things that clearly did not work.
And which ones are those? Knowing what did and did not work is an actual challenge by itself.
> You need a cache, you can get one from any number of cloud providers or spin up something off the shelf you run yourself. Same with databases, CDNs, large scale object storage, search infrastructure, message brokers, and all the rest. So, yes, there might be a need for changing some of that necessitated by some key people disappearing but it's not a massive technical challenge.
The massive technical challenge is migrating existing infrastructure to something off the shelf, then finding and fixing the new bugs in that existing infrastructure and/or your deployment/configuration. That shouldn't be underestimated.
> The massive technical challenge is migrating existing infrastructure to something off the shelf
It's short term disruptive and might involve some fail whales. And then it is solved again. Break it, fix it. Like it or not, that seems to be the plan. If you accept things might temporarily break, it's going to be a lot easier to act.
It's a microblog, not star ship. At least from Elon Musk's point of view, there's going to be a difference between those two and the amount of brain cycles he's going to dedicate to things breaking or not. He's not going to be afraid to break the team (check), the platform (still running fine), or the community (in progress, engagement seems up so far).
These people behave just like the irrational fanboys, except they just do the exact opposite. Being a sheep and being a contrarian sheep are the same thing.
Honestly, I'm hopeful that other coastal tech companies do clean house. I genuinely believe it could lead to a resurgence in the lives of Midwest/middle American states and cities.
There is an absolute vacuum of technology specialists in the middle of the US, because no one wants to "live in the middle of nowhere," and they don't want to earn less than FAANG (MAMAA?) salaries, when half of those salaries can give you an amazing life in the middle of the country (source: my piss-poor salary compared to yours).
Years ago I had a massively downvoted comment when I criticised his AR for CAD vapour ware. As someone who was fully in that area at the time, what he was showing while looking fancy had no practical application in the area of design he was talking about.
Ever watch someone do CAD/CAM modelling? They need extreme precision of input that AR sausage fingers just aren't going to help with. You need a num-pad and a good mouse with a stepped click wheel.
I get it. We share some similar neurodivergence traits. He wants to be right in the detail. Constantly jumping from interest to interest, seeing the hidden patterns and connections that aren't apparent to others. But there are a times when I know I just need to shut up and let someone more experienced talk despite my brain wanting to lead every discussion right into solution mode, or providing additional context mode.
I've spent the last 6 years in management consulting (without formal business education), I agree with him when he says MBAs are useless. We know that the best solutions come from diverse teams with diverse backgrounds, skills and knowledge. Not 5 clones who know how to build value driver trees, not to say the tools they bring aren't useful, but they can be incredibly limiting.
For someone who hates MBAs he's sure going about this take-over like someone who barely passed one (i.e. knows more than enough to be dangerous). Sure, you're hemorrhaging money in operations. You need to cut costs and find new revenue streams.
What are your biggest costs?
Labor. Slash / Burn. The old McKinsey 7% FTE reduction will give you some extra operating cash from the years remaining budget and you know it's not so much that people (in fear of their jobs) won't just pick up the slack to keep everything moving. Do it quick because you need to rip the band-aid off and get rid of all that accrued leave, restricted cash etc. off your books too.
Equipment. Redundancy? Sounds like unused resources we can fire sale.
Contracts. Renegotiate? The only two meaningful levers are price and quantity. Start cutting quantity now, renegotiate price later.
This is all dummies guide stuff and tends to go terribly in reality when implemented all at once all together.
For instance, research has shown companies that lay-off when under pressure end up underperforming against the ones who chose not to.
Now who's going to help build and operate those new revenue streams?
Quick fixes for a quick buck and a whole lot of extra risk.
And Twitter's problem are nowhere near technological. The site needed to make more money, not reengineer the whole thing while advertisers are fleeing because Trump is back on on a whim!
What choice do they have? The company was hovering around profitable (2021 would’ve been without a lawsuit settlement) but that was before they were saddled with a ton of new debt. It’s possible that they could find new revenue models - that pay for checkmark scheme isn’t it but maybe a smart businessman could come up with a better variant - but that takes time and has to be done carefully since they have to keep the lights on in the middle. Driving away advertisers before finding that new revenue source doesn’t leave much time to iterate.
Debt and equity are just different ways for investors to invest in a company. If company isn’t generating any return on investment, keeping the lights on is not sustainable.
I say everyone can join for free and gets one post a day. $8/month plan includes 10 posts a day, and unlimited for $20. While I have no interest in Twitter, I'm told it's quite addictive, so make the first taste free and then reel them in once they're hooked.
If that's it's business model then sure, that's how it works.
A "de facto public square" would be public in conception, construction, and support from the start, which is one of the ways we know that Twitter is no such thing. Though it would likely also have some rules for how speech is/isn't conducted.
And all things considered, advertiser-friendliness is a sort of low-resolution but approximate passable democratic mechanism for marking boundaries of civilized discourse.
> advertiser-friendliness is a sort of low-resolution but approximate passable democratic mechanism for marking boundaries of civilized discourse
This reminds me that progressives have historically always supported corporations as complex hierarchies, scientific enterprises, run (ideally) by “experts.”
Nowhere was a claim forwarded it was progressive or ideal. The claim is more or less that advertisers have some of the same concerns that elected representatives do because consumers have something like a vote.
But then again, if you're scare-quoting expertise, maybe that's not the conversation you're here to have.
I'm scare-quoting expertise because it seems like you think there's an expert way to run a corporation.
And that's a very progressive sensibility towards corporations, and precisely why they actually like them (what's more appealing to a progressive other than a huge centrally-planned organization run by credentialed experts) despite claiming otherwise.
LOL. If you think that progressives are generally friendly towards corporations then you've never actually talked with a progressive, have you?
And corporations are not generally progressive in outlook either, there are too many values higher up on ladder of concerns.
Advertisers are one subset that bring a rough approximation of democratic to their decisions, knowing that each person in the market they hope to reach will be deploying something like a vote with their dollars. As with any democratic approximation, it's only progressive to the extent that population is, though it's poor compared to other democratic mechanisms.
You can take issue with whether democratic decisions are good decisions, of course, but that's likely to be an unpopular opinion for obvious reasons. And hey, I hear the person currently running twitter recently went so far as to say vox populi vox dei. Was his expertise in running corporations part of what you meant to scare quote, or is your challenge to expertise, shall we say, selective?
Read up on Progressive history and, yes, it's very clear that the early 20th century Progressive movement loved the concept of a corporation.
While most aren't self-conscious enough to realize it now, they still love corporations as evidenced by how they zealously defend them when they serve progressive ambitions and power.
That depends on Musk’s goals. If his goal is to make money to pay off the massive amount of debt he gained as part of buying twitter, (as the attempted quick roll out of twitter blue would indicate), then yes he probably needs to care about what advertisers think. If he just wanted to leverage his position as one of the richest men in the world to ensure that twitter was a haven for free speech and screw the profitability, then no he wouldn’t need to be guided by advertisers. He probably can’t both want immediate returns on his investment and to quickly rock the boat though.
>while advertisers are fleeing because Trump is back on
[citation needed]
CNN's ratings were never better than under Trump. He's fantastic for advertising. So is Musk. All controversial figures are. That, oddly, isn't controversial in advertising.
>on a whim!
He created a public poll, and when people voted for Trump to be allowed back won, he unbanned him, tweeting "vox populi, vox dei" ("the will of the people is the will of god"). Had he unbanned him despite the poll saying "no" you could argue it was a whim, but that isn't the reality we're in. He also refused to unban Alex Jones, citing exploitation of child deaths and a personal story. Not unbanning Alex Jones was more whimsical than unbanning Trump was, factually speaking. Why do people always misrepresent his actions? And why is it always upvoted and not flagged here?
I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.
I don’t have personal experience in this, so obviously I can’t speak with any authority. But I have heard from colleagues that tons of little factors can dramatically affect user engagement. For example, even a couple dozen milliseconds of longer load times can push a noticeable number of users away from your app.
Undoubtedly some people will be put off, but think of how often Reddit used to go down -- still got pretty big. And Twitter already has all the newsmakers people want to see. If your goal isn't necessarily user growth it makes sense
I have personal experience with this. The metrics (as much as I despise using them as a source of truth) undoubtedly show a very strong positive correlation between better load times and user retention.
true. if twitter, Facebook, reddit, and hackernews go down for a couple days it wouldn't affect me at all. if GitHub and npm went down I'd me mildly annoyed but could still work.
We had an obituary for Fred Brooks on here just the other day. I'd suggest that his thesis in The Mythical Man-Month conflicts with your comment above (that reduction in staff count for a software project has a good correlation with the ability to maintain it / evolve it / innovate on top of it).
I've never heard of (or thought of) your interpretation of the corollary to Brooke's Law, but removing people from projects until they succeed and are on time seems like a bold strategy.
Happens regularly in the Free Software & Open Source world, where software projects are often forked by a single individual. Obviously enterprise software is on another level, but the principle remains that reduction to a smaller development team (for some period of time) does not necessarily correlate to a threat to viability and has often indeed been a reinjection of vitality.
IBM is a "tech" company that employs 282,000 employees, and when was the last time they invented something? I don't remember the last time I heard IBM in the news about something they made.
The bigger the company, you often times find less innovation and more administration & bureaucracy.
The reason startups can survive is because of its small size that makes it very flexible and adaptable to chaos and change, that gives it the edge over bigger companies.
Homeostasis is a good metaphor, but it implies a living, dynamic system. Something that resists entropy by itself being in a state of flow -- the matter constantly changing, while maintaining the form.
In modern software environments, the entropy is almost violent -- the changes in all the constituent dependencies are constant and relentless. Something frozen in time does not stand a chance, unless it's entirely stand-alone and dependency-free -- an unlikely scenario with a service of Twitter's size.
Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!
>Mr. Musk is also considering shuttering one of Twitter’s three main U.S. data centers, a location known as SMF1 in Sacramento, which is used to store information needed to run the social media site, four people with knowledge of the effort said. If the data center in Sacramento is taken offline, it will leave the company with data centers in Atlanta and Portland, Ore., with potentially less backup computing capacity in case something fails.
There are currently three data centers, hosting all of the real-time and most of the batch production load. Ad-hoc load and some production batch load was migrating to GCP, but this was being significantly curtailed in the months before the acquisition closed because it turned out to be very expensive, much more so than anticipated and more than the equivalent workloads had cost in the data centers.
What did you make of Mudge's report regarding resiliency of data-centers?
> Insufficient data center redundancy, 59 without a plan to cold-boot or
recover from even minor overlapping data center failure, raising the risk of
a brief outage to that of a catastrophic and existential risk for Twitter's
survival.
Thanks! There was a lot in that and I didn't follow it closely. Honestly that always confused me about the dc failing prep. Like I get that it would be a good idea to have a documented plan but at the same time, we did lose a whole DC and figured out how to recover(not that it was that easy) Im not sure having a cold boot prep plan would have helped that much. The things that failed that I dealt with personally during that event, I don't know if I would have foreseen. The site went on afterwards so it didn't seem to be existential.
It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did.
All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point.
Maybe it was close (in reference to external factors not implied in your very insightful post!) but its amazing how confident people are with opinions on events they have 0 insight into. Everyone knows how to solve the war in Ukraine or world hunger but how rare is the "consensus" (in terms of up-votes or popularity) right... just something that got me thinking. Thanks again for this article. I always love seeing details of tech ops!
That's what I have been trying to tell people. There are literally two alternate realities and the extreme polarization tries to suck you into either one or the other. Because what people don't realize is that worldviews are tied to group identification.
My theory is that the economic problems have been stressing the systems and that makes the "bones" more apparent. The core concept of this country is checks and balances of two opposing sides. I think it was a great improvement over authoritarianism, but you can see that it is definitely not the ideal final paradigm.
A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.
I would use it as a case in point. You have one group claiming that Twitter is completely on fire and soon to be closed, and the other saying it's never been better and there was zero fallout from rapidly firing thousands of people including a lot of engineers.
Neither of those extreme viewpoints reflected reality accurately. There were significant problems, but they did not take the entire site down or anything. In fact, for millions who did not have 2FA or use it at that point, they did not see any issue. Whereas people that wanted to find an issue could go looking for the movies. So each side was able to find ways to reinforce their alternate realities.
I think both sides are correct. Anyone with experience at the scale knows that if everyone stops working, the site becomes more reliable. Holiday freezes demonstrate this. But holiday freezes also demonstrate the other thing: if left alone for too long the systems start to rot, from memory leaks and cache ossification and other things that usually aren't noticed during active development.
Personally, I doubt that it will be anything technological that ends Twitter. It will be economic. Their advertising revenue has been decimated and their operating costs have never been higher.
that's surprising actually. I'd expect my sites to run for about a year if i suddenly died. maybe longer. certs will renew themselves, servers will restart themselves. everything is on autopay.
the thing to kill it would probably be api breaks by 3rd parties.
maybe disk space would kill me eventually too.. i don't have auto resizing for my sql database but its got a long runway yet. or god forbid my site grows on its own and the qps kills it.
I mean, twitter is incomparably more complex than your website (I assume you don’t sit over google or Facebook), plus I’m sure elon made people poke at the system (“we only need 20% of the mucroservices”…)
It's hard to really say whether that was due to something Elon did or if it was just a normal outage from code deployments, hardware failures, etc. Anything that might seem like an outage will be magnified now because everyone will think it was Elon's doing and it will get tons of clicks. The reality is that these huge sites have a dozen small/medium outages almost every day, but almost no one notices.
The QRT system doesn't work (if a post says it has 200 quote tweets, you can always open it and never see any), but then again, it never did seem to work.
It's still entirely believable the site will go down and remain down for a week or two at any time.
The extent of the 2FA issue I saw - which, by the way, was w.r.t. sms-based 2fa login, not other 2fa methods - was a single screenshot making this claim.
I wouldn’t be surprised. There were definitely a few bugs. I think ppl unhappy with Elon overplayed things though and now it will look silly.
Our expectations were anchored by the idea that Twitter would cease to exist. So very real problems like 2FA going down look trivial compared to the former.
I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.
It doesn’t bother me because it’s lascivious, it bothers me because it’s the comedy equivalent of fart noises. It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into. Whatever floats your boat, I guess.
I would say fart noises are the comedy equivalent of fart noises, but in any case, I sure do love people getting upset by the comedy equivalent of fart noises. Speaking of which, I would actually pay Musk $8 if it meant he would post a video of himself doing the old underarm fart noise thing and I could see all the fantastic "well I never" reactions from everybody who got upset by it.
>It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into.
What's wrong with the new owner of Twitter using Twitter in exactly the same way that a lot of other Twitter users use it for, memes and jokes?
I find that rather refreshing, actually. It's a variant of eating one's own dogfood.
It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.
His process tree has only produced private and reusable rockets, space internet, electric vehicles, and hundreds of billions in value. But he makes crude jokes like 90% of the men on the whole planet. Clearly there's something deeply wrong.
So if the people HE hired did those things, doesn't that mean HE put the team together, HE found the talent, HE set the vision, goals, and tasks, so HE led them to accomplish these things, along with funding them? It also means HE didn't design the SDS you claim to be defective, almost like such software is difficult to develop.
Is it strange and inappropriate if most people think it is a funny joke and laugh it off? Who goes out of their way to look at Elon posts just to be offended?
It is funny. I think it's hilarious and the fact he shared it in public makes it even more so. It's one of his most liked tweets, so a lot of other people also thought it was funny. Comedy is subjective, but if a lot of people think something is funny then it is objectively funny. And it is perfectly appropriate for public figures to reveal something of themselves and have some fun in public. Puritans will always be offended by this kind of sex joke, or by the gays being allowed to marry or whatever, but we laugh at them too, they make it even funnier.
In my opinion what would be inappropriate for a billionaire CEO, and reveal deep flaws in decision making process, and the decision making process of any government official or governing body who subsequently partners with that CEO, is to fraternize with the likes of Epstein after he was convicted, e.g., https://www.nytimes.com/2019/10/12/business/jeffrey-epstein-.... But I guess morality is also subjective so people might disagree with me there too.
He can get away with posting just about any silly thought because he has no board of directors to answer to and no one can fire him. Big difference when you are thinking of him as a CEO.
Lots of people are puritans, I was talking about the puritans making themselves terribly offended by that meme. If Musk is one too it doesn't mean they aren't, it's not based on some relative scale here. But I am using the lowercase puritan which probably doesn't fit him.
I didn't know that he lead an extremist cult though, not that I have ever cared to find out much about his life or beliefs. It wouldn't surprise me, I think a lot of the ruling class and ultra rich are terrible people who we should not be living under. That said I have noticed the claims of Musk's terribleness and extremism may have a tiny correlation with his unwillingness to toe the Democrat party line. Now quite possibly that's just a coincidence and there's absolutely no causation there, nevertheless if I were ever in a situation where that information would be important to a decision, it would behoove me to take it on its merits rather than on faith.
I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people became really invested in the idea that a site cannot possibly stay up without dedicated SREs, as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations.
I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now:
1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that have 0 people left.
2. More than the technical issues, betting that Twitter will go bankrupt is the easiest bet one can make. Musk saddled Twitter with a shit ton of debt - even if things worked as they did before he had to cut tons of people due to the debt burden.
The issue I see is that #2 directly works against #1. Musk has said it will be lots of intense work adding new features to try to raise revenue. But making a ton of changes, probably with lots of shortcuts to get them out the door quickly, especially when so much institutional knowledge has walked out the door, will make keeping the site stable even that much harder.
"Chapter 7 of Title 11 of the United States Code (Bankruptcy Code) governs the process of liquidation under the bankruptcy laws of the United States, in contrast to Chapters 11 and 13, which govern the process of reorganization of a debtor. Chapter 7 is the most common form of bankruptcy in the United States."
Airlines get free government money though. Will twitter? If not, Twitter will need to borrow more money and at higher interest rates compared to market rates if they go through with bankruptcy.
Funny thing about the bankruptcy hypothesis is that the owner of Twitter is the richest man in the world and could literally pay old twitters expenses for decades even if it never made a penny. If we are just talking about the debt payments, he has a 200 year runway.
Actually be probably can't. Musk's wealth is largely on paper, as Tesla stock. Which is enormously overvalued and has been plummeting since he bought Twitter. If Twitter needs cash injections, he has to sell more Tesla, which will depress the price further.
A billion dollars a year of stock sales is a huge amount of guaranteed supply to the market. And it's non-optional: it has to be a billion dollars a year. But Tesla isn't worth anywhere near it's current market capitalization either.
I think that share quantity and value are very separated concepts. At some point Elon loses control of Tesla and that's it: Tesla's price has been tanking, and it's still on its way down to a sane valuation.
3 Renault Nissan Mitsubishi Alliance 7,680,014 (same) -1.3%
4 Hyundai Motor Group 6,667,085 +1 +5.0%
5 Stellantis 6,583,269 +1 +5.2%
6 General Motors 6,291,000 -2 -7.9%
7 Honda 4,121,000 (same) -6.5%
8 Ford Motor Company 3,942,000 (same) -5.9%
9 Suzuki 2,763,000 +1 12.9%
10 BMW 2,521,514 +1 +8.5%
Notice who's not in that list?
Now let's look at market capitalization[2]:
1 TeslaTSLA $568.87 B
2 ToyotaTM $202.86 B
3 PorscheP911.DE $101.44 B
4 BYD002594.SZ $86.39 B
5 VolkswagenVOW3.DE $84.30 B
6 Mercedes-BenzMBG.DE $69.28 B
7 General MotorsGM $57.48 B
8 BMWBMW.DE $57.39 B
9 FordF $56.60 B
10 StellantisSTLA $48.87 B
There is no possible way Tesla is worth more then double what Toyota is, while shipping about 1/10th of the vehicles. The Price-Earnings ratio is 56, vs Toyota's 9.
By every possible metric, it is vastly overvalued compared to it's profitability, and there's no reason on the horizon to think it will suddenly catch up since all those automakers are moving directly into it's space.
If you want to compare companies by 'how big' they should be, you need to compare enterprise value, not market capitalisation.
If you think Toyota's stock (or bonds etc) will increase in market price relative to Tesla, for a fee financial markets will let you bet on that conviction and make money, if you are right.
It's borderline a meme stock at this point. Tesla is valued as if it will forever be able to sell every car they make, on earnings calls Elon blows off any questions about demand. In the last few months though used car prices for Tesla have plummeted, I think they're finally about to run into demand issues as they get more supply online
I also want to add a #3: the crew that's left is probably on-call 24/7. My thoughts are with the poor souls on that rotation (if your team even still has a "rotation")
It’s quite bad for the business as a going concern and its ability to serve customers and employ employees when it suddenly, and for no obvious reason, takes on tons of debt. Money isn’t free and loans demand payment.
In Twitter’s case, its $13B in new debt on the balance sheet means that, every year, they have to come up with $13B times the interest rate in additional revenue and/or reduced cost merely to be in the same place profit/loss-wise. Elon already massively overpaid what Twitter’s business-as-usual can generate even before accounting for that $13B; as a result, the post-acquisition Twitter has to try a bunch of negative-expected-value moonshots in the unlikely hope that one of them hits against the odds and the others don’t turn out fatal, because doing nothing or iterating sustainably kills the company via debt service.
In a lot of cases the best solution would be for the company to declare bankrupcy, reorganize, and discharge the debt by convincing creditors to take pennies on the dollar and a share of the resulting smarter-run healthier company, and a judge that the plan is reasonable. However, Elon both poisoned that well by firing people, angering advertisers, and bumbling around product-wise, and also staked a bunch of $TSLA that would need to be liquidated to go through with the bankruptcy.
Yeah, it's private equity 101. I can't believe it happens. A bunch of MBA sharks get loans to buy something with some chop shop plan or other ruthless scheme to carve up or spitshine a company.
The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR money back with a profit, leave the debt to the company and the sucker who buys it, and look for the next victim/target.
Musk likely went the rapacious route because he made a dumb move, and this is the only way he's getting the money he committed to the deal back.
Oh yeah, and when they buy the company, they are in charge of it. So they can pay themselves whatever they want in executive bonuses / etc.
Why is that possible or legal? It feels like a loan for purchase of a company necessarily should belong to the purchasers of the company and be paid by the sale of the equity they bought or profit. This way just feels… rigged.
The people who buy the debt know what they are getting into. (And if those 'suckers' don't know, honestly, they shouldn't be investing in junk bonds etc.)
It is not a bad thing as such. The main problem comes when a business is stripped of all its assets and/or weighed down with more debt than it can repay and fails as a consequence, leaving employees out of a job and communities without a formerly productive enterprise.
It is more that it tends to beat raw deal for the purple who lose their jobs for no good reason. The Twitter people will likely mostly land on their feet but most leveraged buyouts aren't affecting highly compensated employees with highly in-demand skills.
> Yeah, it's private equity 101. I can't believe it happens.
Why not? It's pretty good for all the stakeholders that have a say.
The old shareholders sold for a nice price.
The new ownership didn't have to pay for the whole thing.
The lender gets to charge a pretty good interest rate because there's a good chance of default. If they're lucky, they get repaid; if not, maybe they made enough in interest to make it worthwhile; maybe when it defaults, they'll be able to make something worthwhile out of the wreckage they got at a nicer price.
Leveraged buy outs aren't great for stakeholders that don't have a say. Employees usually get new terms worse than the old ones; in this case, there's been some severance at least. Customers get a promise of a big bang bankruptcy in the near to medium term, rather than a slow fizzle. Sometimes companies with a large payment they can't make can restructure, and sometimes they shutdown with little notice. As a private company without public accounting reports, there will be a lot of guessing about revenue and debt service.
Which goes to demonstrate the very sorry state of our society, a society where employees are not (anymore) part of the "stakeholders that have a say" group.
Because at the end of the day, as you mention, it's the employees and their families that will suffer the most. But as long as those employees don't have board seats while strikes and labor-related physical protest movements have become a thing of the past then I guess this is the reality we'll have to live on for the foreseeable future.
No he did not. It was initially proposed but there's no margin loan against his TSLA holdings in the final deal. For the debt that Twitter took on in the buyout the collateral is the company itself. There's zero connection to Tesla in the actual financing.
To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...
Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to.
So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building. But eventually people are going to start turning on stoves again… and then…
This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining.
Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.
Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.
I interpreted the GP's comment less as a moral claim ("the DMCA bot is good") and more as a claim that the DMCA bot's failure is a strong indicator of internal instability (given that it sits directly at the intersection between Twitter's profit interests and microservices architecture).
Put another way: being unable keep a little bot running, one that keeps an entire industry happy, doesn't bode well for other components of the service.
It seems self-evident that the bot was considered low priority, since it isn’t working anymore. But nobody is disputing that: they’re saying that the fact that it is low priority does not bode well.
If it was a prerequisite to land $100M ARR from all the media properties’ marketing budgets to advertise the multi-billion dollar pipelines of the movie and entertainment industry, that lil’ bot was the gate to $11,415 per hour of revenue at risk if its uptime failed to sufficiently please the attorneys and auditors from those customers.
I mean, does Twitter want to be a party to a copyright lawsuit? If not, following legitimate looking DMCA notifications (and legitimate looking DMCA counter-notifications) and responding to suponeas as necessary gets you an affirmative defense for copyright infringement.
You may not like it, but having a bot do that probably saves a lot of legal hassle.
Hm? Intellectual property is explicitly carved out of 230, and even if it wasn’t: it isn’t user generated. Content providers are regularly found liable for infringement on their platforms, especially when the plaintiff can demonstrate willful negligence (which in this case would include discontinuing a seemingly effective scanning system.)
> Nothing in this section shall be construed to limit or expand any law pertaining to intellectual property.
If section 230 from the CDA of 1996 provided immunity from copyright claims, there would be no reason to include procedural requirements for processing claims in the DMCA of 1998.
Many jurisdictions take an even harsher line when it comes to being complicit in intellectual property abuse. We saw this famously with The Pirate Bay, Napster etc.
Well, Disney won’t care why their copyrighted material is publicly available, noone likes this sort of copyright, but if Elon wants to avoid huge fines he better (make someone) fix it ASAP.
Given that Twitter already offered premium API access, they've got billing in place, so now they add a new form that, once your credit card is verified, flicks a boolean on an account that was previously flicked by another process.
It’s more than that. For one it’s not an existing boolean, there are now two different kinds of blue tick that are presumably stored separately. Blue is also supposed to give the user fewer ads (while making them more relevant) as well as additional weight in feed ranking algorithms. It’s also intended to be offered worldwide which adds a lot of complication to things like payment flows.
I’m not saying it’s going to bring the site down tomorrow but that one feature touches on a lot of services.
Clearly worldwide payments is an issue otherwise they’d have rolled it out worldwide day one, and they didn’t. There must be something holding that back.
Plus I really don’t think you can compare B2B payments for premium API access to end user payments. Not least because they aren’t going to be going the same route: a huge number of them will be via Apple or Google in-app purchasing. Ask anyone who works with those systems, it isn’t a quick plug and play job.
In general though, a new subscription tier, feed algorithm changes, UI changes… if these aren’t, what is a big change in your book?
Definitely stealing this as the right way to frame the issue. What would normally be a small kitchen mistake turns into no longer having an apartment complex.
Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.
Whether that will spell disaster when there's a fire depends on whether the building had too many fire extinguishers to begin with and whether the boss can buy new, better fire extinguishers to replace some of them before there's a fire.
> Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.
If we're deep-diving it'd be closer to say that he rolled in and sold 80% of the stuff, largely sight-unseen, and if a fire breaks out he'll find out how much of that stuff was fire extinguishers.
He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's scared off Twitter's main source of income while saddling it with significant debt.
As an SRE I would have been shocked if Twitter failed catastrophically (well moreso than broadly disabling authentication) in short order. However failure is pretty much inevitable at this point given the damage that E-Lon is actively doing.
Well, define 'failure'. Minor outages like the one you are talking about were happening from time to time long before Musk bought Twitter, and it even suffered long outages frequently - remember all those fail whales?
I meant that there will be no catastrophic failure that will permanently (or even for a few days) stop Twitter from working at all.
failure. noun. with no security team, hackers are able to get in easily.
everyone's DMs leak, all the anonymous accounts have their identity revealed, and all Twitter's clients (advertisers) have their bank account info made public.
Yeah, I don't think anything remotely close to it will happen, unless some of the fired developers have left themselves some backdoors which they'll give (sell) access to it to someone.
does this stuff happen automatically? is there a robot that goes out and reads about all the new zero-day exploits and patches all the software without human intervention?
On Hacker News, everyone's a comedian! And, yeah, as the sibling comment pointed out there've already been failures as a result of some musky action. While Twitter isn't likely to fail on its own, E-Lon is actively causing problems. You need people to deal with that, and even if he had motivated, relevant, and competent engineers… how long will they stay motivated without a paycheck?
Let's not forget that whatever code monkeys are left are now personally liable for running afoul of the FTC. Whatever motivation they may have now will run out pretty damn quick once they stop getting paid.
Pretty sure that the developers that are left will not be liable for anything unless they are knowingly participating in criminal activity such as criminal negligence that is the direct cause of someone getting seriously injured or killed.
Generally speaking prosecutors want to target the highest level individuals responsible for directing such activity in the first place, not low level implementers who have little say one way or the other.
For now I'm talking about the FTC consent decree, so administrative penalties not criminal charges. Musky fired the folks who were responsible for ensuring compliance.
How do you propose Musky does that with no payroll department? E-Lon walked back the mandatory return-to-office policy last week. Surely if he could find (or thought he could find) suitable replacements he would be pretty comfortable demanding RTO.
I'm not really familiar with how they do payrolls in USA, but I'm pretty sure it isn't some rocket science (pun intended) and can be done rather cheaply by an outsourced firm.
Also, why are you (and many others here) refer to Musk as "E-Lon"? Is it supposed to be a derogatory nickname?
It's simply an abbreviated version of Elongated Muskrat as far as I'm concerned. Payroll is easily one of (if not the) most complex systems at any company. It's not just statutory stuff but personnel stuff as well. With Twitter you're not just dealing with 50 states and the feds, but with every other company in which Twitter has (had?) employees. There's a cottage industry of payroll firms precisely because payroll is so obscenely complex.
Even if you outsource it you'll still need people within your company to manage your service provider. At one company I worked for they got all of their outsourced HR+payroll for free (indefinitely) because the provider (Gevity) consistently fucked up everything they touched. This was at a company of like thirty people.
If you're suggesting Twitter can simply outsource payroll, sure. But you do that before you fire your whole payroll department. You still need people to handle the transition.
Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent.
Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable!
To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave.
As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free.
Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on.
And the house didn't fall down within 3 weeks of it being purchased.
Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own.
And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."
It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees.
It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail.
There are a lot of places where the systems would start to fall over if 70-80% of the team departed. Especially since a lot of folks left on bad terms and/or were suddenly terminated. It was the opposite of a smooth handover.
So it wasn't unreasonable to think that Twitter would begin experiencing problems.
I didn't think that Twitter was going to literally have an unrecoverable system crash and permanently shut its doors, but I thought we'd see some outages or partial breakage over time, which is basically what has happened albeit in an admirably mild way.
It comes off as a kind of desperation, as
though they need Elon to fail.
You don't need to pathologize it, like it's some... deep weirdo psychological yearning. Some people think he's a jerk and wouldn't mind seeing him fall on his face!
If a person was staking some significant part of their emotions on their feelings for Musk, yeah, that'd be unhealthy.
But I think you are significantly overestimating the emotional weight behind 99.99% of the half-baked Twitter quick takes. It's okay to not like a guy!
But the parent poster characterizes those disliking Elon Musk as suffering from "psychosis" and "desperation."
It's a an unfortunately common, passive-aggressive ad hominem tactic you see on the internet and elsewhere. "People don't like thing or person XYZ? Oh, they must be mentally ill and losers who spend their whole lives obsessing over that thing/person"
> put their reputation on the line saying Twitter wouldn't survive the weekend.
Was that a real thing those people were saying?
Genuinely asking, as I don't follow any of the former Twitter engineers on, well, Twitter, and if there were any such posts/articles on here I must have missed them.
What is with all the people this year who insist that because something doesn't happen right away, it can never happen?
The only prediction anyone made was that the World Cup was historically a period of very high load, so if something was going to go wrong soon, the weekend would be the first vulnerable time, and the World Cup finals will be the next.
Many of us have worked at companies where there is a lot of duct tape holding things together and when you let go of entire teams (not just a large percentage) then it isn't unreasonable to be pessimistic. Especially when you know that in order to fix problem A you need to take B, D, E, C corrective actions in that order. And you learnt that through years of things going wrong.
More so at companies like Twitter where they never really reach a steady state. You constantly have large fluctuations in system stress e.g. World Cup, Trump rejoining etc.
Just curious, among the critical stuff just holding on, was there also a whole pile of departments and teams doing work that could cease tomorrow and the company would blink and move on?
My experience is both exist at the same time because the leadership teams don't actually know what core business is or are busy building empires and resumes.
Just curious, among the critical stuff just
holding on, was there also a whole pile of
departments and teams doing work that could
cease tomorrow and the company would blink
and move on?
I get what you're saying. For any given team with a public-facing product you generally have perhaps 20% of the staff keeping things running and the other 80% of the staff is working on new features, reports, enhancements, customer support, whatever. You could eliminate them and while it would diminish the company in various ways, the other 20% could certainly keep the lights on.
However it's worth noting that's not what happened at Twitter; there were very specific and explicit reports that the "keep things running" teams were hit just as hard by layoffs/resignations as the other departments.
So there was real justifiable concern there.
There's also a lot of things that can go wrong during sloppy and abrupt handovers. Like... you fired the guy who manages the domain renewals. In the chaos of transition nobody picks this role up. One day 18 months later you realize "twitter.com" has expired. Or whatever. Even if the remaining staff is sufficient to keep things running, there are thousands of these little process interruptions.
I don't even doubt that they did a lot of work, but as this article suggests, that's part of the reason why we _wouldn't_ expect the site to just go down in a couple days. If all the servers needed constant manual massaging that would speak more poorly to their work than the other way around.
I will admit to this. I was so angry at how Musk treated his employees, and I took them at face value when they predicted outages. I still think that we will see outages in the coming weeks and months, but probably not in spectacular fashion.
A lot of Musk’s fans get emotional about this too. I remember people gloating when they rolled out the new Twitter Blue, saying that Elon got more done in a couple weeks than Twitter had gotten done in years beforehand. And then they shut it off, and now they’re delaying the relaunch. People were praising Musk’s “moderation council” concept, and then he promptly abandoned it to make these decisions by fiat (and in one case, a poll).
I love this comment. It's so salient. It's probably hard for a lot of people on this board to digest but, I feel like most know where you're coming from.
The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.
> all software goes down and requires specialist intervention eventually
Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional?
If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right.
If there are 100 critical systems running at 3 9’s, you’d expect 2.5 hours of downtime per day.
So yeah, all software should keep running. But it doesn’t. And something like Twitter isn’t “a software”, it’s a very large assembly of lots of different software systems and the exponential math that dependencies create.
Yep, and when one of the SEVs rolls around that would have been small (say 5m of downtime fixed with a flag flip), it instead will have a nontrivial chance of escalating into a major multi-hour/multi-day outage without the right institutional knowledge.
I'd guesstimate that Twitter probably has dozens of services that are in the critical path of an average user interaction. It's hard to keep even logically optional dependencies truly optional in large scale systems involving many people.
However Twitter didn't die in the past when fail whales ruled its day, so they probably won't kill it now. It's just not that kind of business. (In contrast, a one hour outage had me directly apologizing to our largest customers on the phone). That said, Twitter can only be unstable and lack feature growth for so long before something else takes its place, so Musk is on a clock.
Right, but Twitter wasn't a healthy business (in the sense of being profitable most years) in the first place so it's not beyond the realm of possibility they took reliability further than made sense. Anyway they now have a huge debt load that changes the calculus regardless.
I had MySQL running on some bare metal for many years without a restart.
I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully.
Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.
Even if your mysql instance and hardware had run indefinitely, if a table is being written to it will eventually run out of disk space or key space and crash. How long it will take depends on the application but it will happen eventually and if no one is around to fix it...
> I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.
Anyone who has ever been oncall can intuit how often stuff breaks in big or little ways. Sometimes it's transient and goes away, sometimes it can be filed away to be fixed in the next year, but sometimes, it turns out to be an all-hands-on-deck crisis for a team, or 5.
> The default, naive assumption should always have been programs keep running indefinitey on their own.
...for people who understand software to some extent. I get the feeling a lot of people see it more like a hamster wheel, where once the developers are gone it immediately starts noticeably slowing down as it stops (and are confused when that doesn't happen).
Now, if your Rust code was a distributed system that handles spiky loads from ~330m users, and processes petabytes of data, then I'd consider your comparison relevant to Twitter.
But I'm going to assume it's not relevant.
P.S., I've written Java services that never went down, because they had a well defined domain and all potential errors were handled. But, I'm not about to compare that to all of frigging Twitter.
The infra usually matters way more than the code. RAM or a disk will typically fail before the Linux kernel, and it's written in the boogeyman language.
It wasn't only news outlets. A majority of my... politically-noisy tech friends on Facebook went through a recent phase of intensely posting about Twitter on fire and collapsing.
Now just because they're "in tech" doesn't mean they have any idea about Twitter, but they should at least know enough to know they don't know what's going to happen, but obviously they're not actually using their brains when posting comments like that. Point is a lot of people opposed to Musk have been participating in a spiraling echo chamber of fairy tales and wishful thinking, it's not just journalists (although clearly they're printing lies with ulterior motvies too, as usual).
I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office.
e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.
I personally have somewhat mixed feelings. I do enjoy a good dumpster fire like any good internet denizen ( and I won't deny that there is an odd level of.. excitement maybe.. that one major social platform may go down ). I have no love for social media as some of my posts may indicate and for a variety of reasons including the amount of oxygen this creation takes and the amount of confusion it causes in general population, it would not be the worst thing ever.
That said, the sheer amount of weird emotion attached to this particular billionaire is just perplexing. Only few years ago he could do no wrong. Today, it seems, he is being ganged up on including by some of his biggest internet fans. And media oblige for whatever reason. I dislike billionaires, but I dislike them as a caste in our societies. I don't see Elon as a savior of mankind or the devil, but those emotions that swirl around him are made to make him look like one or the other.
Somehow, one is supposed to pick sides.
Trump seems to create the same emotions. It is truly fascinating.
I was about to say that you're talking about two different problems. But, no, what's happening to Musk is the same thing that happened to Trump. Their inner circles knew how rotten they both were, but outside of that they were largely unknown.
New Yorkers despise trump and have for ages. From tenants, to contractors, to business partners they all know what a shitheel he is. Outside of NY though trump was largely seen as an affable caricature of a rich dude. Case in point: NYC as a whole isn't averse to republicans and yet they still rejected trump in 2016.
Elon's the same way. Sure he called that diver a pedophile. But he also had a Tesla employee hounded until he fled the country. Musk had the guy swatted for fucks sake (or tried to at least). Or what about the journalist who pissed him off and ended up having some Tesla features disabled on his car. Or his ex-wife who talked about being treated like an employee. These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight. Now the whole world can see that he went full on Howard Hughes.
<< Outside of NY though trump was largely seen as an affable caricature of a rich dude.
This is a very interesting statement to me. I am trying to recall conversations with my wife's side of the family, who supported Trump ( initially, mostly ) and their impression of him was almost exactly that ( plus, "he is a businessman so he knows the game" or something to that effect ). I am in in Chicagoland, but now I am curious if any serious study was done on whether that is how he was perceived as ( ideally before it became socially awkward to state your political affiliation openly ).
<< Their inner circles knew how rotten they both were, but outside of that they were largely unknown.
And that is the other interesting thing. Prior to whatever caused this change of hearts, I only saw impressions of Musk that were similar to impressions of Gates ( you know the ones - saving Africa or something; I know it reads awful, but I really don't remember what his PR presents him as ) - visionary making cameos in odd cultural vignettes ( I think he did one in 'Big Bang Theory' ). The articles describing his work style were very slowly bubbling up.
<< These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight.
And became too much of a power center himself.
I will need to sleep on it, but I think I agree with you.
It's because even though Elon Musk is no right-winger, he voted straight Democrat his entire life until very recently but then he voted for a Republican. The haters will brook no deviation from their accepted political narrative, and when Musk did it the hordes of people with pronouns in their bios turned on him.
For those people it has nothing to do with technology, which is why their takes were so hysterical and wrong. It's 100% about politics every day, 24/7.
The bubble where the cronies who helped plan it were huddled together in the Willard to watch the show put on by the people they directly interfaced with and who were equipped to breach the building.
Some of our friendships are not based merely on political affiliation. This is not a shot, but please understand that not everyone is automatically looking for a clan and an enemy to bond over ( even though it is not uncommon and perfectly rational thing to do based on our evolutionary path alone ).
Not to search very far, me and my coworkers discuss politics ( and we do have somewhat divergent views ). The bubble only exists if one does not talk with other people. In other words, other people being like-minded is not an automatic pre-requisite.
Yes, I've heard it all before. People who were drawn in to the baseless Russiagate conspiracy theory, and all the other ones have said exactly the same thing to me as you are now. Sure, I'm a secular-heretic for wanting to see evidence for extraordinary assertions from people with obvious ulterior motives, rather than taking them on faith.
So far, the conspiracy theorists have had a poor track record as far as I can see. That said, I don't engage or try to debate my friends on it, they have always become extremely upset by any debate on such topics. And they're not hurting anybody so my philosophy is just to leave them to their faith. I don't go around challenging people to prove the existence of God whenever they make a religious Facebook post either, I'm not that kind of asshole.
> Yes, I've heard it all before. People who were drawn in to the baseless Russiagate conspiracy theory
As you can read in the Republican-written Senate Intelligence Committee report, everything about Russiagate did in fact happen and it was 100% real. (except for the Steele dossier/pee tape)
This essentially comes down to Trump realizing that if you say all your crimes in public, nobody will think it's a crime and it won't be reported as one, because you don't "seem like you have something to hide".
I might be basically be picking at it with what I am about to say next, but was it not admitted that 'Steele dossier' itself was a rather elaborate disinformation campaign[1] as noted by Fiona Hill? I really don't be want to be this guy, but if we are going to have that kind of discussion here, we might as well discuss current state of knowledge.
edit: All this is relevant, because that dossier itself was the basis for 'Russiagate'. I am ok with a rational counter-argument.
> As you can read in the Republican-written Senate Intelligence Committee report, everything about Russiagate did in fact happen and it was 100% real. (except for the Steele dossier/pee tape)
The conspiracy theory is that Trump colluded or conspired with Putin to hack the election. No evidence to support this has ever been produced. I know that may be difficult to accept, and I ascribe no malice to those who genuinely believe conspiracy theories, they are a powerful psychological trap.
When Trump told Russia to release his opponent's emails they'd hacked, and they did release them afterwards on the same day, he got away with it because he just did it in public on stage, and so cynics can't accept it because it's too obvious - it wasn't done on an encrypted Signal chat in a smoky backroom, so it must not be a conspiracy, so it must've been a coincidence.
Plus his campaign manager was working for him for free and instead being paid by a GRU agent for their targeting data.
You can try denying the Senate/Mueller reports on this, but you should read them, since they do say this.
The joke he made at a political rally about Russia releasing Clinton's emails if they had them? That simply is not not evidence that Trump colluded with Putin to hack the election. Calling that evidence is like a climate denier calling an unseasonal snowfall evidence that global warming is false. And that's not what the misinformation peddlers at the time were claiming to be the evidence, either. Remember Adam Schiff's repeated false claims of "ample evidence" that he could not share but alleged were shown to the House Intelligence Committee that he chaired? That was the kind of inflammatory misinformation that drove the whole conspiracy theory.
This is how conspiracy theories hook people, they tell people what they desperately want to be true, and they falsely claim there is this vast amount of solid, air-tight evidence, and it will all soon come out. That gets people invested, hooked, convinced that it is true. When things eventually get disproven much later and it turns out there is no such evidence forthcoming, the believers are so invested in it that they are incapable of accepting the reality, so they cling to whatever flotsam they possibly can -- "Trump did collude with Putin to hack the election, therefore this joke at a campaign rally must be a message to Putin, therefore it is irrefutable evidence that Trump colluded with Putin to hack the election". It works on probably a subconscious level, it is a very difficult to step back and see this when you are in the hole, let alone dig yourself out of it.
No report into this including the vaunted Mueller Report has ever produced any evidence that Trump colluded or conspired with Putin to hack the election. I have read them. Mueller Report On The Investigation Into Russian Interference In The 2016 Presidential Election, Volume I, p. 181, second paragraph:
For that reason, this Office’s focus in resolving the question of joint criminal liability was on conspiracy as defined in federal law, not the commonly discussed term “collusion.” The Office considered in particular whether contacts between Trump Campaign officials and Russia-linked individuals could trigger liability for the crime of conspiracy—either under statutes that have their own conspiracy language (e.g., 18 U.S.C. §§ 1349, 1951(a)), or under the general conspiracy statute (18 U.S.C. § 371). The investigation did not establish that the contacts described in Volume I, Section IV, supra, amounted to an agreement to commit any substantive violation of federal criminal law—including foreign-influence and campaign-finance laws, both of which are discussed further below. The Office therefore did not charge any individual associated with the Trump Campaign with conspiracy to commit a federal offense arising from Russia contacts, either under a specific statute or under Section 371’s offenses clause.
Citizens United case means that, by design, there will be no evidence of money transfers from foreign powers to US elections.
so the idea we can 'find no evidence' is because people rigged the system so there would be no evidence.
Democrats could start taking money directly from Xi Jingping or from some communist in Venezuela and we would never know about it. Republicans can take money from the Saudis and we would never know about it. This is the reality of modern politics in the US.
infinite invisible money and zero evidence that anything happened, by design
Right, if you're already deep in the hole and all-in on a conspiracy theory, there are endless ways you can explain away holes in the theory and turn conjecture or coincidence into hard evidence. Even the lack of evidence can be evidence to people taken in by conspiracy theories.
Same as how China secretly paid the liberals to create the global warming hoax, that's another one along the same lines as the Russia collusion one that some people actually believe.
I actually find this fascinating and and am not taking sides. If Jan 6th was literally an attempt to overthrow the government, why hasn’t Trump been convicted of treason? I thought swift action would be taken if the slant in the stories news media have reported is accurate.
But more importantly, why can’t you be friends with people who have different politics? Honestly I’m not sure what word best describes a world where everyone has to think the same to get along, but I have to think it’s far from ideal.
Dems didn't want to throw him out. Pence was a competent politician. Trump wasn't. They wanted him embarrassed, not out. Much like the Brits had a plan to assassinate Hitler before D day. Choose not to. They were afraid he would be replaced by someone competent.
> as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations
Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different and have different time to resolution.
Riiight. But what happens when there's nobody on that oncall rotation? I've worked at FAANGs, and I'm just thinking what would happen if there was a problem with an upstream team and nobody there. Maybe it's ours now. Maybe there's a principal engineer who worked on it back in the day that's still on the payroll? Maybe we're just going to try to bring it back up with no idea what is going on? What if you have a bug that silently loses data?
> but it was never going to happen overnight and I'm not sure why news outlets were saying that
I don't think it's that obvious, it's one thing to just leave everything running but Elon was also talking about changes he was making (Ex. turning off a bunch of "microservices" because they don't do anything). If you turn off the wrong thing and don't have anybody left who knows how to properly turn it back on again, then you're in a pretty bad situation. It doesn't seem like that happened, but I don't also think we have enough information to say how close it was to happening.
This whole thing has been quite telling for me. I don’t use Twitter and am not a Musk fan, but seeing someone try something to move away from ad driven seemed like a good possibility for me.
Then I saw how bitter and nasty a lot of the online communities I belonged to were. I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue. I was met with people attacking me as an idiot with clickbait articles of “proof” that ad models were irrelevant to twitters problems etc, as if I was doing anything other than quoting musks official reasons for the purchase. Suddenly Twitter was this great beacon of graceful discussion and Musk has ruined it.
People just love to be mad, no one had anything nice to say about Twitter and in a flick of a switch they’re holding completely opposite opinions. We’ve always been at war with Eurasia, it seems
>I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue
In the computer industry, there are some famous historical examples of companies that announced their new product before it was ready, people stopped buying the old one, and they went out of business before the new version was done.
Not quite the same business, but a bit reminiscent.
Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.
Right. It’s not like if all the engineers walk out it’s suddenly going to fall over. It’s that when an issue comes up there may not be the expertise on hand to fix it. So the remaining twitter engineers should expect a rough time over the coming months.
You assume that small issues are ever prioritized in the first place. My experience with large tech organizations is that their priories are driven by internal culture, not best practices, just like any other company.
Although I haven't worked at Twitter, if it is anything like the places I have worked at, the small issues get ignored because nobody makes their career on things that don't show up on investor / shareholder / upper management radars. If it can get papered over so it ends up being someone else's problem in the future, what actually gets worked on is flashier things like redesigning the timeline AGAIN so users spend more time on the website or (in the case that caused me to work for small startups again) spend three weeks deciding on a super trivial design decision with 12 stake holders.
There must be some plan for this, though, right? If it were me and I were trying to do more with less I'd plan to cut some features (for instance, are Spaces that critical to the operation? Because it seems like running them would be demanding) and try to migrate things to managed services to reduce the operational load.
I don't have a DR plan for "mad billionaire buys the company", no, but I do agree with the OOM reaper logic - I start cutting the services that use the most resources that are least necessary.
I don't mean that they had one sitting around, but anyone who's stayed with the company has to find this their most urgent priority, I would think, between the focus on cutting operational costs and the simple fact that they have far fewer people around.
Well, that was a bit of a joke, but I have been around the block a few times and been on one side of a merger or other, and gone public to private and back (as an IC/soldier, never been management). The Twitter acquisition is the craziest thing I've ever seen; there's a reason it's big news.
These kinds of things have formal courtship type periods and legal filings that take months, and then they happen, and then executives plan on rolling out layoffs and cost cutting measures over years, not days or weeks, like the "plan" seems to be here. This really is madness, and I can't blame people for thinking the site might just wobble and collapse at some point, though as the author goes into in very nice detail, we do try and build resilient systems. I'm certainly curious how it all plays out, but you won't catch me making any guesses.
This kind of schedule and chaos looks more familiar for acquisitions of a small company I think. I guess it is strange for it to happen to a company of Twitter's size, but I think the big-news aspect of it has more to do with how much journalists use Twitter.
> I guess it is strange for it to happen to a company of Twitter's size, but I think the big-news aspect of it has more to do with how much journalists use Twitter.
I think you hit on the the main point: the reason this is constantly in the mainstream media is because the mainstream media so heavily relied on/still relies on twitter[1].
If this were any other large company imploding in a Musk acquisition we'd see fewer stories about it because it affects journalists less.
--------
[1] It's going to be especially funny if journalists complain that "#LearnToCode" type of tweeters aren't getting banned, and get told in response to go off and create their own twitter.
No matter where in the political spectrum one may lie, it's always satisfying to see a group being fed their own lines back.
> This kind of schedule and chaos looks more familiar for acquisitions of a small company
That's certainly possible and I wouldn't know what that would look like, and yep, in theory there are lots of financial rules and regulations regarding public companies that Musk made a bit of a hash of.
I can't agree that Twitter's journalist demographic was mostly responsible for this being big news, though I'm sure that plays a part. Musk brings the circus with him wherever he goes, but this was nuts even for him. And any $44 billion tech buyout would be news without those factors.
software, like everything, is subject to laws of physics
I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.)
Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" and change. For the preservation of their own self-interests, of course; much like the company that makes physical objects too robust and runs out of customers, planned obsolescence and the desire to change things and justify it so they can be paid to do something are still there.
It's possible to make things which last. Unfortunately, much of the time, other economic considerations preclude that.
If software ran without side effects, perhaps. But it doesn't. Databases grow, files are uploaded, logs pile on, messages and events propagates and filesystems fill up. This is why entropy matters.
Exactly. Tiny memory leaks in seldom called functions can also cause slow degradation over time. People wonder why a simple restart seems to 'fix a boatload of problems' but this is often the reason why.
> I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.)
Sure, those binaries might work the same when executed. Although the probability of that is never 100%, but as you pointed out, the rules of arithmetic aren’t expected to change any time soon. That’s correct. Unfortunately software does not exist in its own micro-verse, it’s subject to the laws of physics acting on the machines it’s running on. So while you might be able to write scripts that work decades later, it’s much harder to ensure those scripts consistently run for decades. RAM chips, CPUs, and everything in between are guaranteed to eventually fail if left running unsupervised in perpetuity. Entropy raises with complexity. At Twitter’s scale, to run a software service you need globally distributed cloud infrastructure. They likely have hundreds of services, deployed to many hardware instances distributed across the globe. Twitter isn’t 1 script running 1 time producing a single result. It’s hundreds if not thousands of systems interacting with one another across many physical machines. Layers of redundancy help, but ultimately cascading failures are a mathematical certainty. Many would argue the best strategy to reduce downtime on these systems is to actually optimize for low recovery time when you do fail.
Software is also bound to the world in other ways. Similarly to how most business processes, products and even more generally, tools, change over time, so too do the requirements placed on software systems made to facilitate or automate these things.
Ultimately the only way to escape the maintenance cost of software is to stop running it. The longer you leave a software system running, the more likely it will eventually stop.
Even if the entirety of Twitter.com were mathematically proven correct, it still would run on servers that are made of physical bits that are subject to entropy.
It’s possible to make things that last if you are in total control of the whole stack, including hardware.
Embedded systems that still do their job after 30 years do exist but they live in isolation in a specific and controlled environment, and are built for a limited, unchanging task.
On the other hand, complex web software is build on layer upon layers that are not in Twitter’s complete control.
Hardware change regularly, requiring changes at the lower levels of an OS, inducing potential changes in behaviour, performance, which require adaptation as a consequence.
And that’s before considering security, eternally moving goalposts. Not just at the OS or network level, but also at the business level.
Twitter and al are not living in a locked down context, they live in the messy world of human interactions and that alone requires constant tweaking.
So yes, a binary is more like a mathematical construct and by itself it won’t rot, but if the world around that binary changes, you need to change the binary as well, and for that you need maintenance. The amount required depends on the complexity, brittleness and how well your stack is engineered, but implying it’s a con is a bit extreme.
Computation is literally bound by entropy. Math has no such limitations unless you explicitly define them.
I thoroughly recommend researching entropy as it regards to e.g. information theory, systems engineering and even (perhaps especially) to machine learning.
Computation is ultimately about what we can compute _in this universe_ and the forward flow of time is an emergent property from the universe’s innate entropic guarantees.
Time is “pre-sorted” for us thanks to entropy, enabling us to define algorithmic complexities over the time domain in the first place.
> we don’t think so. the prod incident we heard about involved someone making an ill-advised choice to reactivate a large account, causing a huge load on the social-graph system, on the night before a prolonged high-traffic event.
Spot on. Absolutely hate this attitude that software sitting there just gathers wear and tear as if it's a mechanical device. Software is written with a particular target platform in mind: x86, ARM, Nvidia GPUs, FPGA soft-processor etc. If the hardware you are running on doesn't change, your software should still function. If the specs of that target platform don't change, your software should still function. If the specs of the target platform change but a hard-working compiler engineer has done the work to make sure your software gracefully uses the new features (for example, a compiler optimizing using AVX instructions), your software should still function.
The fact that most software doesn't continue to function even on the same platform, and on the same hardware, is a massive indictment of the software industry's standard practices.
Complex software has complex failure modes though.
An application running on a single platform, self-contained and with some basic failovers such as redundancy (2+ machines running the same application), etc. should have ridiculous uptimes.
A distributed and complex system with interdependent components, under variable load, with different capacities for subsystems running across some thousands of machines will, inevitably, encounter some unforeseen state which degrades the system as a whole. It can be a small component that cascades a failure mode in an unexpected way or can be a big component failing spectacularly due to a bug, or race condition, or a multitude of other issues that are not entirely predictable and guarded against at the time of writing such software.
The latter is what has "wear and tear", it's not one software, it's a whole system of software communicating with each other in varying states of hardware decay, you can design and build it to be resilient against a multitude of predictable issues but you can never expect that it will run perfectly fine unattended.
Unfortunately it’s not this simple because most non-trivial software is written with a dependency tree, every node of which may discover vulnerabilities (or performance problems) which, when patched, trigger update cascades in this tree.
You're forgetting that software on it's own is basically useless. In order for it to provide value, it has to be operated by a physical machine. All running software is physical, with spinning disks and mechanical relays, electrons being pushed back and forth, and photons flying around. Twitter is not a piece of software, it's a complicated physical system. software is an abstraction.
Don't forget that users have been trained to be incredibly fault tolerant as a result of how flaky general software can be. Now that cars are having BSOD that tolerance may reach new levels or just evaporate.
Thanks I agree, I think small issues can build up over time, itll be interesting to see what happens. Wish I could see the future post mortem docs from the outages
Turning it off and then back on again probably fixes the issue. There’s very unlikely a grand ticking time bomb just waiting to bring it all down. Recycling servers will probably keep it running.
> Turning it off and then back on again probably fixes the issue.
Turning a large scale system entirely off and on is never simple. Invariably you’ll run into some kind of circular dependency that must be manually investigated. And even tracking those down becomes tricky.
Classic examples are things like DNS, service locators, or authentication systems. And large tech companies are notorious for NIH-syndrome for all of those.
There’s so much redundancy built into modern distributed systems that you can reliably bounce a VM without issue. You can reliably roll bounce a series of VMs.
Twitter doesn’t have unique scale problems by todays standards.
First, Mastodon isn’t the only option - it’s not like both couldn’t fail just as both AIM and MySpace did.
Second, Mastodon runs on open protocols. That has good and bad points - for example, it won’t grow as quickly as a project with huge corporate backing - but it does mean that there’s a more direct link between the community and its longevity. Twitter isn’t just flailing because Musk is doing management by bong rip but also because he’s desperately trying to get out of a financial hole. Open source projects have different kinds of financial challenges but they’re never on the hook trying to fill a hole measured in billions of dollars, either. Given the number of communities older than Twitter I’d say it’s far from proven that Twitter will outlast anything.
I found it really interesting that one of the main Iceland instances is being run on a Raspberry Pi out of some student's living room in Sweden. Over 500 users at the moment.
https://types.pl/@tritlo/109383888427885539
why? a single app running on a single server is several orders of magnitude more resilient than a spaghetti clusterfuck of services upon services. twitter could be brought down by a single expired certificate
- completely dead with ssl invalid certificates with expired domains.
So you would have to keep moving to another mastodon instance (if you're lucky) or try and run your own instance and join the many instances with the three issues above.
There is no monetary incentive to keep a mastodon instance running and we both know that begging for donations doesn't scale.
There's a gigantic flood of new users (i.e. literally multiplying the userbase) in the past few weeks, so yeah, a lot of servers are restricting signups to cope.
I don't think "begging for donations" needs to scale, if instances get too large to keep running then smaller instances should (and do) split off, they can still talk to each other after all.
There's a pattern of diverging expectations here, one is the non-technical/naïve one,
- Twitter is going to go down tomorrow and it's all over. RIP.
The second is,
- Twitter is going to experience a failure cascade over time.
The third is,
- It's all going to be fine.
I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps?
I am not that good of a developer, but watching Twitter I can't help but be reminded of Arecibo, except at a larger, more abstract scale. There was no single massive event that caused the failure, rather a series of factors and events, tiny cables breaking that eventually leads to a failure cascade that then causes the suspended platform to crash.
From what I can tell, in the past week or so,
- Twitter's copyright system failed
- Two Factor Authentication broke down (it seems to be back up?)
- (anecdata) Tweets have been loading sporadically for me and other people, sometimes we try to open a tweet and it says that it doesn't exist. Happens more frequently with new/recent tweets.
- (unconfirmed) Twitter's managed account backend is behaving "strangely." For e.g., "One of my campaign managers logged in last week and found all our paused creatives from the past 6 years had been reactivated." from https://www.teamblind.com/post/i-told-my-team-to-pause-our-750kmonth-twitter-ads-budget-last-week-4dnbo1Ft ———— Friends have told me other similar stories
Are these failures symptomatic of a larger problem, or are they well-isolated parts misbehaving? Can Twitter even experience a failure cascade like Arecibo? Can that be paused/stopped?
I am asking this question because I don't know. And I'd like to develop a better mental model to understand what happens next.
Also anecdote, but twitter has historically been pretty unreliable for me.
Without a reliable twitter systems status history pre-acquisition, the reports of failures, like the issues with the 2FA system, don't mean a whole lot.
Definitely this. I regularly see people complaining about ancient bugs as if they were new after the acquisition. Nope, you just are engaged enough to notice them now.
After the failwhale days, Twitter has been quite stable for me with the only exception being the live-refresh features on tabs of the website that have been open for days (which I don't think many websites would handle well).
There has been a serious degradation in the quality since the acquisition:
- Sporadically loading tweets - I could go on some tweets and refresh the page multiple times, with tweets fading in and out of existence showing "This tweet is unavailable"
- Tweets that quote tweets of accounts you have blocked behave weird in multiple ways. Sometimes it's just showing a "This tweet is unavailable" instead of "This tweet is from an account you have blocked", and a few interacting with them crashed my timeline on mobile, having to restart the app
- On a few occurences, every third tweet on the timeline was an ad
- 1-2 notifications from crypto spam bots reaching me every day. The same thing previously was filtered out quite reliably (I assume), since then it happened ~once ever 6 months
--------
And those are only the things I've seen personally. Yeah, they are no deal-breakers, and mostly sporadic failures, but it very much feels like a service that is degrading by the day.
Perhaps the failure mode is like the Tacoma Narrows Bridge, with Elon as the wind, the undulations his whipsaw changes of mind, and the employees the hapless cars being flung into the river.
When I was learning programming, I "wrote" an internal app where a supervisor input some information in a form, it connected to an oracle database and then printed out an outbound manifest for a truck full of car parts.
I put "wrote" in quotes because what I had actually done was install Apache and PHP Nuke on a Windows NT server and then modified an existing form page to do what I wanted.
I wrote that application in 2002 and never had a problem with it. I never restarted Apache or the server or did any maintenance. I didn't even upgrade Windows NT if I'm being honest. Windows NT became unsupported in something like 2006 and I left the company in 2010.
I received a call in 2018 from the last person still working there who knew who I was. It had finally fallen over and they wanted to know if I could help. That was the first time it ever had an issue.
Twitter pretty much passed it's first real test with flying colors since Elon took over.
The world cup is on and the site didn't collapse.
That is a huge win and while I don't believe Twitter will ever be profitable, I think Elon will be feeling rather smug right now with his skeleton staff.
I've sent Elon an email asking for internship, though sadly I can't move to Sanfran and live in the office floor for 12 weeks like @geohot can. Still...
Is this based on personal experience, or are you going by some well known publicly available stories about Musk?
I've heard pretty much every possible opinion about the guy, but the ones that impress me the most are that he loves to go deep into technical topics with people who know their stuff, regardless of their credentials. That, to me, is a sign of a man with deep humility and desire to learn.
At the same time, he's not considered a very diplomatic person and evidently is rather hard to get along with both in his personal life and professionally. Yet, his accomplishments are substantial.
So I would go with maverick, different thinker, fk your feelings, kind of guy who not only isn't afraid to break a few eggs to make an omelette, but probably isn't even aware he's breaking the eggs :)
The Mesos detail struck me as a risk for the reason you stated.
Also, the article seems to suggest Twitter only has two datacenters. That seems surprising for the global reach of the company. Perhaps there are other smaller datacenters that are not prepared to handle the entirety of the site’s traffic.
My current thinking is there’s time to figure out how to operate the current system before it runs into issues that would render it degraded for a prolonged period of time. I noticed TLS certs have already rotated for instance. That was my best guess for simple thing that could fail if managed poorly.
The company was moving off it. I wish I could find a twitter eng blog about it. Interesting there's a bunch of not directly from twitter sources about the decision though.
Twitter didn't start yesterday. There was a time when Mesos was all the hotness and k8s was this new thing that looked promising but wasn't nearly production ready.
Apple is also a big Mesos user, but also moving to k8s.
They were one of the first users of Mesos, but they didn't create it.
> Mesos began as a research project in the UC Berkeley RAD Lab by then PhD students Benjamin Hindman, Andy Konwinski, and Matei Zaharia, as well as professor Ion Stoica.
> The social networking site Twitter began using Mesos and Apache Aurora in 2010, after Hindman gave a presentation to a group of Twitter engineers.
The article is good, and informative but a little odd in one respect. It uses air-quotes to introduce concepts like "rack" and "shelf" in a DC, but does O(logn) notation mid-flow.
If you don't know what a rack is, how are you meant to know what the Order of scaling function means? Thats a highly computer sciences specific notation, and if you grok O(n) you know what a rack, a host, a DC is.
Hah fair point, I was trying to write for a more general audience. Part of blogging for me is to improve my written communication skills. Appreciate this!
Something like "scales linearly" or "scales exponentially" probably covers it, for whatever case it was (I forget)
really good article btw. really enjoyed reading it.
might pay to flag when systems you talk about are twitter-internal or are open-source. People love that kind of thing. "wow: twitter uses the gnu C compiler" type thing.
I get your point that both of these are specialized notations/terminologies, but it's entirely possible for somebody to understand algorithmic complexity without knowing what "a rack, a host, a DC" is, there's no inherent connection between the two. As somebody whose interests tend to be more theoretical than practical, myself prior to reading the article would be an example.
Very interesting article. But the constant misuse of “to” for “too” kept throwing me out of reading mode. I wonder why my brain does that instead of reading past it since I know what is actually meant.
Tangent but I do hope that Musks trim down results in orgs that have less “executives” and a layered cake of a org structure, and more autonomous small teams that execute on shared overarching initiatives.
I really don’t understand why so many tech companies have like 8 layers of engineering levels. If the argument is that you need more money so more levels, just have a bigger band. Don’t chase titles they don’t mean shit. Not to mention the management stacks that seem to just hang out in meetings and take pvt. I haven’t worked at a proper startup but I’ve been on projects where a dozen or so people rebuilt apps used by tens of millions of people in a few months, or launched completely new applications for bigger companies.
Now that I work in a tech company (not big tech, but still a multibillion dollar corp) I’ve noticed that since IPO we have added a ton of bureaucracy whereas back in the day we were small teams building completely new and at times complex features. Literally was in a meeting earlier this month where people were patting each other on the back because we added a single attribute to a table. I’m obviously reducing the entire initiative to a small thing but it kind of explains all we had to do. It’s soul crushing but with the economy the way it is I must deal with it. Hell I’m down to show up next Monday and work for Elon if he wants a go getter. At least I’ll get to DO stuff. Or any other startup in SF if they are hiring.
A lot of those upper layers and the bureaucracy which is seen as "bloat by the bottom layers comes about from compliance. The compliance burden grows as the business grows (e.g PCI, insurance and government requirements kick in at certain thresholds).
Sure engineering adding a new widget to the site might increase profits by 12%, but all that bureaucracy can prevent the company losing its payment provider or breaching a government regulation which might cause the company to close overnight. So if the stakes are +12% vs -100%, who is actually doing the most important work?
I don't disagree with your desire to work somewhere lean and task-focused, but I think it's almost impossible for that to happen anywhere but small(er) workplaces.
I work at a Big Tech adjacent (or Big Tech, depends on your definition) company and was there pre-IPO and it happens for the reason that varjag described in their sibling comment. What makes it even more idiotic is, as the pre-IPO/lean culture dilutes, more people will use their level to pull rank in meetings, as the level itself becomes more of a target than the work/goals. Then the politics around levels will become ultra-competitive. Google didn't get to their tortuous promo process in a vacuum after all.
If 10 engineers can build 5 features in a week, then why can't 50 build 25? We easily understand why as engineers on HN, and even managers and investors understand why. But that wont stop them from trying to solve that problem. And some of them do! All of these big FANG companies are as big as they are because at some point they figured out how to solve this problem. But just like time and growth break architectures and software, they also break processes, and they eventually encounter a level where they can't solve it anymore. Then they stagnate, then they either die or solve the new problems. They have to grow to keep investors, and to grow they have to keep investors. That's why it doesn't stop at 12 dudes building an app. But sometimes it does just fine. It's really just a matter of who's investing and what returns they're looking for.
> Tangent but I do hope that Musks trim down results in orgs that have less “executives” and a layered cake of a org structure, and more autonomous small teams that execute on shared overarching initiatives.
The core problem is that this approach doesn't really scale out because communication overhead exhibits quadratic growth to the orgs size if it's untamed. The feasible options are:
1. Let the complexity bring chaos across the org
2. One decision maker rule them all
3. Gives some sort of management structures to the org
Your proposal is somehow between the option 1 and 2. The option 1 works pretty well for smaller orgs and it might scale to a quite sizable business if the members are generally competent so org-wide trust can be well-established. But anyway you'll hit a road blocker eventually since people cannot spend all of their time on communication overheads. The option 2 just moves the burden of entire complexity into a single personnel so it's not really a reproducible solution but more of a mere luck.
Hence the option 3 is the only remaining option for regular orgs and many smart people tried to figure out the best structure (or at least best practices) but unfortunately we don't have a definitive answer yet. Google-style "tech level" is one of the tool to reduce communication overhead by setting a common structure for expectations (e.g. "we have 1 L6 and 3 L5s to take that project" is generally easier to convince than length explanations of your team members). It's not ideal but it somehow works so it's adopted.
You're likely right that you'll be much more productive if you can get rid of those bureaucracies, but getting other folks convinced is a completely different story. Trust takes time to propagate and people have a limited time to spend on it. This obviously could be drastically simplified if you can work with Elon (or similar style leaders) directly but his time is extremely limited so there will always be only a small number of people who can enjoy that privilege...
> If the argument is that you need more money so more levels, just have a bigger band.
Someone earning a lot more than you at the same nominal position leads to a lot of resentment: the perception is that you are clearly wronged here. On the other hand a rank system makes this less objectionable and offers at least some roadmap to a similar income. "Oh, she's SWE L9000 naturally she'd earn that"
I wonder how big Twitter's infra costs are (capex). Based on the article, they need to build capacity > 2x peak loaf to tolerate one of the DCs failing. But if they instead had 4 and still only tolerated 1 failure, they'd only need capacity > 1.33x peak load.
This doesn't account for the extra overhead associated with extra DCs, but it seems like there's opportunities for major effeciency wins.
You can usually do more with less if the remaining few are up to the challenge. It will come at some cost, like having to "move fast and break things" more. There can even be a positive side if the previous situation had a lot of friction due to "too many cooks," as I've experienced on several teams. What's more worrisome is the alleged 80hr work weeks. Yes, two SWEs on a team aren't as good as one SWE working double hours, since SWE-time is known not to scale that way. But burning people out, aside from being unethical, is risky when nobody is ready to fill the spot.
Musk has a point, but it feels more like he's doing this out of financial desperation.
I'll bet it starts going down a bit more often, nothing too severe. Bigger issue might be inability to roll out new features, especially if they dig a horrible tech debt hole. Maybe someone has more details, I haven't kept up, but it seems like the $8 verification rollout got botched because they cut corners on actually verifying the subscribers.
Feels like one of those "ideas that sounded good in the conference room" that turns out to have unintended consequences.
Hopefully they can fix this and have an effective way to verify identity... or else just give it up and go back to the old, creaky, somewhat effective method since people are used to it.
I wasn't really expecting a large failure quite yet. I'm guessing there are enough people left with secondary or tertiary knowledge of systems to keep things ticking over for now. The question is how long that state of affairs can continue before some combination of worker burnout or deferred maintenance will stretch things too thin. Then, all-hands emergency work to get things running could contribute even more to burnout -> resignations etc.
They could stay ahead of things if they hire into areas related to infra maintenance that got hit the hardest, before issues reach the point of cascading failure. Or maybe put everything not critical to keeping the site up & running on hold as remaining devs get a bit of cross training using the limited people w/ institutional knowledge that remain as trainers.
For myself, I don't see much value in Twitter, in terms of net social value. It's format seems pretty much designed for only the most surface level discussions, part of what I believe leads to some of its toxicity: it's simply too hard to have conversations complex enough to invite enough discourse for that to tip the productive/toxic ratio a bit more positive.
So I've been rooting for Twitter to fail for years. I'm not rooting against Musk: I like his other businesses and at least a portion of their ongoing success is tied to his persona. (At least before the Twitter stuff: His actions there may impact his relationship with major banks that his other businesses will rely on, and Musk & JP Morgan, a bank that wasn't included in the deal & therefore won't be hurt by it, is already on somewhat negative terms with Musk. The twitter deal has added a few more to that list, and other banks have undoubtedly taken note. Some bank will always finance the regular sorts of things any large corporation need for him, but they're all going to be pricing in some additional risk. That's not really a big deal, it's just that I think Musk's persona has previously been a net positive for his companies and now it's lost at least a little bit of that.)
I don't think it's that crazy. Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations.
So now that it is owned by someone that doesn't need churn and just needs to reduce cost, people can focus on discreet resilience factors, just like any small tightly held software operation. Many of Twitter's pet projects go away, the ad sales relationship have to get re-evaluated too, but the core consumer product that everyone sees can be made resilient and operate cheaper.
The problem is that it’s run by someone who overpaid badly and needs to come up with significantly more money to pay for the debt he saddled the company with, and then his actions seriously disrupted ad revenue. That puts him more in the PE playbook of cutting costs as deeply as possible even at the extent of long-term growth. Unlike his other companies there isn’t strong government support to drive business for Twitter.
He's going to sell Tesla shares to pay off his debts, he's knocked off $4 billion already, he'll be able to keep doing that, make offers to buyout stakes of Twitter from people that really don't want to be involved in this shit show. Twitter will operate at lower costs, and also not be profitable for him given current information. That's financially fine.
The banks probably took convertible debt at high interest rates, but who cares about high interest rates when it's only been one month. He probably has a clause for accelerated payments, because its debt, not shares yet.
(after a certain amount of time, or upon default, the debt would convert to shares. but it can be much more complicated, and onerous than that too. lending can be fun.)
Musk has been in a similar position before when he took over Tesla. At that point it was struggling to make the Roadster, they were facing having a marginal cost of manufacturing a car higher than the sales price.
Musk took over and made cuts. I don't think anyone could argue that those cuts were at the cost of Tesla's long-term growth. They were needed because of an imminent liquidity problem. Tesla has hired great people since then.
> Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations.
Did we read the same article, because it seemed clear that all that resilience work was done when the company was public shareholder owned, not under the new private owner.
I read the article, I wanted to use the space to point out that Elon's fortunate that it has been done, and also that the technology infrastructure things he needs to focus on aren't impossibly challenging to do, if people are expecting it to be.
So operating Twitter with 80% fewer engineers isn't the voyeuristic suicide that many of us are hoping to be amused by.
Twitter is a lot of drama, but ultimately this is a common private equity play:
1) take private
2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary
3) spit shine/repackage what's left with a theoretically more appealing balance sheet.
4) resell to another sucker...uh, buyer, or take public again.
Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presumably 5 billion in costs. Unlikely to add revenue in twitter's model, he's cutting costs.
Honestly, at this point, is there revolutionary technology that is needed to keep the lights on at twitter? Do you need graphQL gods and SREs that would ace any Amazon raising the bar and master of silicon valley interviewing?
Nah. He'll honestly probably outsource a ton of the upkeep.
It's ugly, but the Twitter board and shareholders took their money and ran, and abandoned the product and the workforce. They could have backed out and let Elon Musk off the hook of his dumb contract with them, but they just wanted the money, and sold to someone that just wants to get his money out of it too.
There are a few key differences. PE tries not to massively overpay, it'll put more effort into targeted layoffs, and it'll maintain enough of a "business as usual" posture to not scare away advertisers. Edit: they also put in a CEO whose full-time job is being CEO of that company.
But yes, this is very bad execution of the PE playbook.
Elon thinks he's brilliant at everything he tries his hand at, so he thought he'd jump into the private-equity arena and show everyone how brilliant he is.
4) who'd that sucker be? How much hubris would it take to sell investors on the idea of salvaging a failing company that "the guy who killed the internal combustion engine" and "the guy who made rockets land on their tail" couldn't make profitable? If it comes to selling off a failure, Musk's reputation will be quite a dealbreaker. The days of whatever happens, if all else fails we can still sell to Yahoo are long gone. Perhaps China might be interested, for obvious reasons, but surely not after Twitter has gone down fighting, fading away into irrelevancy in the process.
If Elon holds the line on advertising and drops costs such that Twitter turns two billion in profit, they yeah the narrative in the business circles will be that Elon trimmed the fat and turned twitter into a viable property and someone with cash will buy it. I mean, Microsoft paid 8 billion for github! They bought Skype for 8.5 billion (probably like 12-20 billion these days), and THAT was after ebay bought skype for 2.6 billion (and none of the source code rights) in what was considered a vast overpay.
I mean, they got an offer for roughly a 10x deal. At that point it was in the best interest of shareholders to sell. Especially that it was yet to turn a profit.
Like, I like my car, but would probably sell it for an order of magnitude its worth.
Chinese developers are famous for making apps serving 1.4 billion people, with at most 1/3 salary of US, and the willingness to do 996 a whole year around.
I would be surprised that no company in the west take advanced of that.
The mythical man month exposed a wealth of problems in software development, only some of which have been addressed in the last 44 years (MMM was 1979, right?)
2000 miles and a language barrier do not help things.
SpaceX and Tesla made affordable, reusable rockets, and affordable, high performance electric cars with a long range. Those things have obvious value, but required solving big engineering challenges to make them a reality. Musk doesn't seem to have any clear goal with twitter. There are vague ideas about making it subscription-funded rather than advertising-funded, and about having less censorship, but neither are things the majority of people want.
I imagine the site will mostly continue to more-or-less work, despite all the layoffs. They still have thousands of staff. The network effect of Twitter is so big that people will continue to use it even if fail whales become more common. Others suggest that it will soon crash and burn, or that Musk will get bored and sell it for a few billion. Or that Musk is a genius who will make some sort of amazing Twitter 2.0 that does for social media what Tesla did for electric cars. But without any appealing long term vision, and with an owner who bought it to satisfy their ego rather than with any real plan, the reality may be more boring. I imagine it will just languish for many years, with occasional manufactured drama, and occasional downtime, but no real innovation. Maybe to be eventually supplanted by something else in 10 years or so.
> Or that Musk is a genius who will make some sort of amazing Twitter 2.0
Based on texts exposed via the lawsuit about the purchase, I don't think this is the case. I don't think he, or his advisors, understand that the product at twitter (and other social media) is content moderation. You can have a vision of whatever type of content or pricing scheme you want but without solid content moderation you will lose advertisers, gain lawsuits (people were posting movies the other day) and lose users because the "feed" becomes a muddied mess. Users are only really the product when you can moderate their content to have profit via advertisers.
In every software company of over 50 employees I have seen up close it would be fairly to identify 50% of people that had minimal to negative contribution.
That said, it takes some time to get to know the real workings of a company, knowledge required to select the right 50%.
That said, afaict the current mo is keep the lights on for the current offering while a new team builds a new product for that enormous user base that offers something much more profitable.
Having worked as SRE on efficiency projects for BigCo, it is not at all uncommon to recover more than your lifetime salary in company savings with only a few months work or even less. The scale of things is so immense that even a slightly better handling of things can lead to outsize returns.
Laying off someone like that, rather than putting them on full-time efficiency work, is an obvious waste.
There seems to be a load of people in this thread making statements on both sides that a) the article doesn’t prove there were too many people at twitter b) the article implies there were too many people at Twitter c) Twitter is a simple app why so complicated etc.
I would say this; Twitter at this scale is extremely complicated. We don’t have enough information to know what all these roles were doing and how many were truly superfluous. The article gives an interesting insight into something extremely complicated- it might be a slice repeated in a lot of areas. I’m of the opinion that you can do a lot more with less but there’s no need to send emails to everyone in the organisation being an arsehole and essentially accusing them of being lazy, then rehiring people who never worked there just for the meme.
Anyway Musk will blow up some rockets, put enormous pressure on people to fix them, and come up smelling of roses again even if he treats a load of people like crap along the way. His fans will say he’s a genius and his enemies will say he’s a dickhead. And so it goes.
The reason why Twitter hasn’t crashed is it’s well written and well orchestrated. Once a bug comes in and crashes something that is when the chaos starts. It’s almost a guarantee that the current crop do not know how to fix the bug. It will be interesting to see how they handle that.
> It’s almost a guarantee that the current crop do not know how to fix the bug
Seeing lots of comments like this. Why do you feel it’s ok to say that, when you don’t actually work there or have intimate knowledge of who is currently working there? The pure speculation and nonsense surrounding Twitter at the moment is plain awful.
It’s just statistical probability. The number of employees left is small and the amount of code is huge. It’s not a huge leap to say that a bug is unlikely to be in the area of expertise of the folks that are left. I would say it’s the probability is greater than 50%. You should notice the probability is not proportional to the employees still remaining. That is because I am guessing some good folks are still there.
Once thing I've seen that I think is overlooked in the whole "is Twitter's tech stack inefficient" chat is the nature of the service.
A lot of the "Twitter could be run off my laptop" style comments seem to come from people who run, effectively, Read-Only services. They might serve data at thousands of queries per second but the data itself is _slow_. It is video files or music streams or other data that updates infrequently or, if there is dynamic data involved, it comes from a very small number of fixed sources.
Twitter deals with thousands of posts per second that are subject to huge fluctuations in the density _and_ those posts have to be disseminated to the millions of users. Twitter is a two sided problem.
I've heard this elsewhere, and when Twitter was originally founded in 2006 it wouldn't have been a particularly hard problem.
Twitter processes roughly 10K tweets per second. Even if you bloat out the text quite a lot to account for encoding overheads, metadata, etc, etc... and assume that each one is 10KB, then this is just 1 Gbps. A single NIC on an old server.
Okay, I get it, Twitter needs a lot of data too. Lists of users, etc...
Twitter has 450 million monthly active users, which sounds like a lot, but even if there's 1MB of profile data per user such as who they are following, that's just 400 TB.
That's... not that much these days. A large-but-not-enormous database cluster.
Sure, there's "historical" data, but that can be compressed and put on cheap cold storage, like S3 or whatever.
Give me a few million annually as an opex budget, a small team of decent developers, and I can guarantee you I could whip up a cloud-hosted service that can process tweets at Twitter's scale.
Obviously, what I can't replicate is the much larger set of tools and systems behind the scenes that are used for moderation, analytics, ad sales, etc...
That's looking at Twitter from the other end, as a Write-Only service with non-realtime requirements on the read layer.
Twitter is both sides. Tweets are replicated out in milliseconds. Twitter search is lightning fast, comprehensiveband powerful. o
Old tweets are not just glaciered into S3!
Did the OP confirm Elon theory that most of the stuff is not needed?
> For four of those years I was the sole SRE for the Cache team. There was a few before me, and the whole team I worked with, where a bunch came and went. But for four years I was the one responsible for automation, reliability and operations in the team. I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it. (There might be only one or two other people)
If you only need one person for the caching department (which is, as I understand, is critical as it delivers most of the data); then maybe you need a handful other dozen engineers and there you have a functional Twitter.
That or the OP is full of himself. Kinda like Musk?
I read this as they built the “tools” (automation, orchestration, monitoring, etc.) for this system, not the system itself; which aligns with the common definition of SRE.
Yes that is the normal case. The post was refuting the assertion that one engineer can run these services indefinitely as previously the OP had the help of SWEs oncall and also fixing bugs.
Its possible to teach a horse to eat less and less each day. They just have this unfortunate tendency to die just after you succeed in weaning them off food altogether.
A horse, of course, is not same as a reduction in force.
Engineers don't eat hay, neither does a horse have a 401(k).
For all the hand wringing about keeping the lights on, Twitter more than doubled its headcount in 3 n bit years. I'm not sure anyone knows how much of a skeleton crew is required to keep things ticking along anymore given that sort of hiring binge.
Great post. Thanks for the insight on how mature Twitter’s Cache infrastructure tooling is. Having done similar work at a different place where you’re exposed to the realities of racks and racks of machines I could appreciate the evolution of the stack and knocking off problems one by one.
I suspected that the hopes and fears of Twitter’s quick demise were overstated and shared[0] some of those thoughts last week, on Twitter of course.
I have seen this happen twice at a smaller scale. Turnover is too high quantitatively and qualitatively to survive by hiring new people, so Musk's "solution" will probably be to oursource a lot of stuff to some consultancy firm. Twitter will almost inevitably rot within 5 years. But don't expect anything noticeable overnight.
Edit: also the reason Musk wants to shut down all those microservices has nothing to do with money, it's that he doesn't have enough manpower to maintain them. If Twitter is like most non FAANG out there those "microservices" are coupled with each other, so good luck.
There was a long twitter thread from some one proclaimed 10+ years SRE listing a lot of things that could bring the server down. Lot of them are pretty far-fetched and outdated.
You're thinking of https://twitter.com/MosquitoCapital/status/15935411779656785.... Remember, on Twitter your likes and follower counts are completely independent of your actual skill area and level. Most of that list reminded me of 2009-era site operations, nothing like what operating a high-traffic site is like in 2023.
It's a little thing, but it's the little things that cascade into big things that require teams of people to fix. Here, we see this person who's quite proud of his automations making the simple, but harmless error of using a possessive instead of a plural. Humans are good at reading through this sort of thing.
Computers are not. Somewhere deep in all his scripts is a misplaced ', `, or ’ waiting to break something.
Linear extrapolation seems to be a common theme in the comments. People take a product they know and then essentially say “Twitter only has X times as many users, surely they only need X times as many engineers. The other employees were excess!”
I haven’t worked on Twitter-scale products. But I do wonder if it’s safe to draw a straight line between services that have middling and large scales. Things don’t necessarily work the same at the extremes.
> if you have a cluster of servers where serving 1000 requests a second might cost $1000, you can instead use the cache to store the responses and serve it from that cache server instead. Then you would have a small cluster for $100 and a cheap and large cache cluster of servers maybe for another $100. The numbers are just examples to illustrate the point.
Reddit needs to implement this if they want to get to the next level.
> When I joined the team the first project I had was to swap old machines that were being retired for new machines. There were no tools or automation to do this, I was given a spreadsheet with server names.
Writing a program to store a list of servers to be swapped instead of keeping them in a spreadsheet sounds a bit like buying a brewery when you want to drink 1 beer.
Program used by a team of one sounds like over-engineering.
This spreadsheet was, from the OPs narrative, the master-list used by the whole operations team. An operations team that is now reduced to "ashes" - therefore not a bad idea at all to have your server inventory centralized.
Anybody working in the biz knows about technical debt. At Twitter that will skyrocket now. My firm believe and experience tells me that keeping everything somewhat up to date over time always pays off.
The problem is not that things will break. Things break ALL the time. The problem is more about avoiding cascading effects and the time it takes to fix stuff.
I think it would be more surprising if it did go down immediately.
Presumably in the turmoil nobody is deploying new code. Sure shit happens, but i imagine that twitter is mature enough that they aren't having weekly critical incidents, especially during times where nothing is changing.
I still think twitter is going to have some disaster, but give it a few weeks.
He says at the end his automation allowed him to automate more. A great positive feedback loop. I see the inverse is also true. Where having no automation makes automating new things harder. Without enough engineers I think they will hit some issue and just not have the bandwidth to fix it in a reasonable amount of time
Why do people assume that things would break so easily? I'm pretty sure Twitter (ex)team did tons of scalability testing with loads that are 10x or even 100x. Plus random 100x to 1000x spikes to account for breaking news. And all of these are done automatically, constantly, and consistently over the years.
There will always be those that think that Twitter can be written by a single programmer in a day or so. How hard can it be for such a simple idea after all, given that there are hundred of tutorials producing a version in just about every language just a search away.
Keep it running? One part-time should surely do it?
Well, debugging a company in prod mode like this is adventurous. Just like we remove code to see if things break, and put it back piece by piece until it's not. That's how some of us get rid of bloat. (Yes, I'm aware of orange and apple comparison)
I don't understand why would a mature platform just fall down without its engineering team. There are surely a lot of small fires but 80/20 rule with regards to triaging should keep the lights on when properly applied.
News at 11: people who waste 90% of their work time on HN while collecting a-la FAANG salaries because they are used to bullshitting their management and getting away with it are terrified of the management that can look at the code.
Considering the maintenance cost of a twitter-scale site, it is inevitable that even a very small-scale problem they will experience will become much larger with the snowball effect.
Why would twitter go down due to staff layoffs? The stuff I build would keep running for years without my involvement. Do you mean that other systems are hand cranked?
"Trends" were subject to Twitter's "trends blacklist" before; something that leaked a few years ago. Maybe they're working correctly now that they're unencumbered. Can you describe how they're not working now?
>- Appealing flagged tweets isn't being responded to.
Mine was responded to in a handful of hours. Much faster than I was expecting.
>It's only a matter of time before these get abused with no one to fix them.
Let don't kid ourselves here, even if Twitter crashes and needs a few weeks or even a few months to recover it doesn't matter. People who leaved will comeback again, after a year everything will be like nothing ever happened.
And even if Twitter lost 50% of it users and revenues, it is still more profitable than have to pay the salary of 7000 people.
More profitable than what? It was and is losing money, plus now it also has some serious debt on top. With staff that actually know their shit (and a sane CEO), it may be possible to turn it profitable, but I don’t think it will happen as things currently stand.
automation is indeed key. IRC networks run with no obvious employees and salarys and been running fine since the dawn of the internet. i see Twitter as a centralised, walled IRC. the # was pinched from IRC if u remember.
An IRC network server can serve what, a few hundred thousands of concurrent users’ text messages on a single PC.
Twitter has around 200 million daily user, peaking up to a billion, running on their own server farm. They are not remotely comparable, especially if we note that read-write dbs are notoriously hard to scale.
Has anybody suggested the idea of rewriting twitter in Rust yet to Elon?
Memory safe, needing fewer servers(hence less $), and the story sells itself to someone like Elon who jumps on such impulses...
^ is a snark. I'm not suggesting that they should..
Twitter won't go down that easily. Besides, Elon Musk hasn't bought it for no reason. I'm sure that he has his plans with Twitter to implement new features and whatnot to restore it.
Till now, he's fired a lot of engineers probably because:
a) he doesn't need so many.
b) he thinks he can do the job himself.
Interesting how this summer all we heard from Musk was how Twitter was overflowing with bots and spam, but now that he runs the place he sure spends a lot of time bragging about all-time high "user" counts.
Interestingly, my reports on racist posts are being accepted much more often than they were before, so this is in fact not happening.
The site was also consumed in horrific discourse for the last week (topic being "is making chili for your neighbor racist, ableist and oppressing autistic people?") so all in all it seems to behaving as usual.
Exactly what kind of views do you think moderates were worried about not being tolerated? Going by the definition of the term it seems unlikely that anyone, much less millions, was previously deterred by policy.
He wasn’t banned (only suspended for a few hours) and that was directed at a single trans person. That last part is almost certainly the reason why he got that warning unlike the many people who have generically expressed similar beliefs.
Their previous CEO said they saw on the order of 500k bot signup attempts daily, and the reports last week specifically mentioned layoffs affecting those teams and things like reporting. Put that together and I’d be careful about comparing numbers before and after this month since they’re almost certainly not measuring the same thing.
Catastrophe tourism. Also, contrary to Elon’s shitty graph with cut off y-axis (can we stop doing that?), that’s like a 1% change around a major American event.
The "why hasn’t the site gone down" stuff is the wrong conversation. The engineers built a train engine, and this robber baron held up the train and forced the engineers off. Inertia will keep the train going for a good long while because they did such a good job of building the train.
At some point the engin’s fluids will run out and need topping off. There's a warning system though so someone'll get notified before the engine blows up, so it's entirely possible this hardcore crew of H1Bs pulls it off. (Great job replacing the SSL cert!) At some point the train’s boiler will need retrofitting. A new team could successfully replace the boiler, but doing it while the engine keeps running just isn't an easy job.
The handful of devs and SREs at Twitter will do their damnedest to keep the train, err, site running. They might even succeed. That doesn’t prove the site was over-engineered, it means they built a damn fine train engined that successfully pulled the whole train up and over the mountain pass.
Twitter wasn't run like a lean start-up type business. Because wasn’t one. Society needs better than that. The vulture capitalism mindset is what's wrong with this county. What's the barest minimum I can pay people to work for me? Not; what can I pay them to let them live prosperous lives but what's the bare, subsistence minimum.
America needs a middle class and these people were part of it. You don’t get a middle class by injudiciously paring things down to a not-even-skeleton crew that’s going to be worked till they burn out, only to get fired.
Twitter was only $14 million from profitability in Q2, and any idiot can randomly fire people to find that much in that large an organization, but cutting costs like that is to misunderstand the situation entirely. Just like starving yourself isn't healthy and won’t improve your self image, randomly cutting costs like that saves money, but doesn’t result in a healthy business.
Out of all the comments here, I can't believe this is the only one that shares my sentiment. I'm not a software engineer, but in my area of engineering if I ever revealed anything publicly as detailed as this kind of architecture; for instance, the failure modes effect analysis framework use at a certain aerospace company... or details of the simulation models involving the crash analysis at a certain car company, so soon after departing one of those companies, I'd expect a letter from a lawyer within a week. In most places I've worked I don't think I could even post anything on the internet invoking the name of the company unless it was a repost of some public information the company itself had previously released.
Do the vast majority of people here on HN/in Silicon Valley not have 5-year NDA's? Or do they and they just not care?
Breach of contract possibly. Usually companies don't want you revealing their internal infrastructure publicly unless you get permission. Who knows though.
Nothing here seems to approach anything like truly proprietary knowledge and the kind of information it contains is pretty similar to what goes up on engineering blogs (or what you'd describe in a job interview, if you prefer) all the time.
Yes exactly this. Maybe they did but I didn't find it stated in the writeup, which was actually pretty good. If it were my company though, I'd want the intellectual capital protected and we programmers usually have to sign non-compete agreements so I think my original comment stands, it was my opinion and initial gut reaction, not an indictment or judgement in any way.
To be hair Musk posted much more internal information in a screenshot lately. The question isn't really is it incredibly useful, but is there anything that might be a breach of the employment agreements. To this I don't know, but it is still an area where employees should be careful. I personally would never write anything about my employer on a personal blog.
Who's going to compete with Twitter? It's a company worth billions, and to execute the caches and automatic fallover in the rival company with billions of users it would take billions.
I know most of my comment is going to be redundant, but I need to say it in order to point to it an year or two from now as an "I told you so!"
Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment.
Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loyalty pledge email out of cockyness. He really wanted to kick out everyone who doesn't believe in him and his beliefs. And he sent that email well after understanding how Twitter runs and putting his loyal people from Tesla ans SpaceX in charge of operations.
The woke crowd needs to get out of denial and start coping with the fact that Twitter is not the bastion of unopposed woke ideology anymore.
He doesn’t understand shit, you don’t fire that many people before even having as crude of an understanding as was seen on that whiteboard. Also, people working on embedded systems for rockets know jack-shit about a system serving petabytes of data.
It's a form of progressive ideology that is largely based on symbolic actions and relies on mechanisms like shame and shunning, struggle sessions, etc.
So instead of actually doing any sort of real tangible action to solve problems, you mostly spend your time denouncing your enemies, including the ones in your own ranks that are not pure enough.
It's very much associated with the left-wing parties in many Western countries, who have for the most part traded their historical defense of working classes for this new brand of performative progressive politics.
Twitter will soon experience the negative effect of the mass layoffs and die slowly. If it survives, please don't ascribe it to Elon's management skills. Don't forget the employees who are stretching themselves there to rescue Twitter.
The biggest thing that brings down a site is changes. Typically code changes, but also schema/data changes, infra/network/config changes, etc. As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine. The trick is to design it to be as immutable and simple as possible.
There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site.
The last thing off the top of my head that will absolutely bring a site down over time, is expired certs. If, for any reason at all, a cert fails to be regenerated (say, your etcd certs, or some weird one-off tool underpinning everything that somebody has to remember to regen every 360 days), they will expire, and it will be a very fun day at the office. Over a long enough period of time, your web server's TLS version will be obsoleted in new browser versions, and nobody will be able to load it.