Context: I teach at Princeton and study social media and recommendation systems.
From a very quick skim of the repositories, this appears to be quite limited transparency. The documentation gives a decent high-level overview of how Tweet recommendation works—no surprises—and the code tracks that roadmap. Those are meaningful positive steps. But the underlying policies and models are almost entirely missing (there are a couple valuable components in [1]). Without those, we can't evaluate the behavior and possible effects of "the algorithm."
I work on Google Assistant Suggestions and I don't think it's very practical to open-source an algorithm like that including the models and the underlying data. Both of them can live in separate services and be frequently updated.
I am assuming that open sourcing the code aims to increase transparency about the business logic of the ranking decisions. At the same time you don't want spammers to be able to easily run experiments against a cloned version of your system.
> But the underlying policies and models are almost entirely missing (there are a couple valuable components in [1]). Without those, we can't evaluate the behavior and possible effects of "the algorithm."
Haven't gone through yet, but yeah, if that's the case, all this is, is a glorified framework to plug your own in.. Not exactly what was promised.
So I don't think you did, since you posted so quickly and that's a LOT of code.
I also haven't skimmed this code except very superficially, but perhaps you should since you're out there making statements with your Princeton credentials.
(I posted this comment with the heads-up a few minutes after your comment above and then expanded it as you didn't respond.)
The spam SEO issue should be dealt/thought about _before_ engaging in the whole adventure, and having to guess how it could work if decently implemented properly defeats the "open source" spirit of it.
More credits would be given if the very idea of open sourcing the algorithm hasn't already been discussed to death with predictions of the difficult points and how it probably won't happen in any sane way.
And them be pilloried for not doing it or not fast enough. Damn if you do, damn if you don’t.
I’m starting to think the broblem with Elon is mostly personal, he’s just a proxy and default wrong.
(not that I approve of his behaviors, but I can’t enjoy this whole mobbing that he’s getting; not that he cares this I’m not worried he’s getting traumatized in any way? it’s just how it’s become an identitarian trait for a certain group that irks me.)
Makes me wonder if a way to override people SEO hacking the algorithm is to create a market of open-source algorithms that each individual can choose and then it's not trying to hack THE algorithm but having to hack many and not knowing which algorithm an individual is using.
Yes but right now there is 100% of the users using the one algorithm (or chronological). If one doesn't know what percentage or which people are using which algorithm, it becomes harder to know which ones to try to hack to have the biggest result.
Is it valid to focus tracking a Dem/Rep split when that split is an exclusionary design for many Americans? Or is it not exclusionary in your belief? I'm curious of a social science perspective.
Ignoring the global nature of Twitter for a moment.
Well if they say “we will open source the algorithm” and then what they really open source is a little bit of slightly relevant code that doesn’t allow us to understand the algorithm, then what we can deduce is that they are trying to weasel out of public commitments.
I can’t say for sure if that happened, but if they made a clear promise and then did something else, it’s perfectly reasonable to call that out.
Devil's advocate though: imagine you were to open source (probably with quite a short deadline) some 'algorithm' used in whatever you work on, but the rest should stay private; how would you go about that?
I don't think it's easy, there's inherently some interface(s!) where it's a hand-wavey 'get the thing from the private bit', and defining that sensibly is hard, and if you try to do it well will probably lead to a lot of meetings, scope creep, etc. - and as far as that goes it's not easy anyway, since it's highly technical and implementation-specific yet also a management/policy decision to make.
It depends on what your goal in open sourcing is. Are you looking to provide a base for others to build software on, and to provide a way for others to contribute back to your code? Then publishing the code makes sense.
Are you looking to build public trust in you and your organization? Then dumping a bunch of code with no context isn't going to help much, as it's not code but behavior that builds or destroys trust.
Are you looking to lean into a polarized partisan environment, pushing a narrative where its you and your supporters against an unfair group of "others"? Then a big splashy move high on symbolism and low on substance that will inspire lots of high profile, divisive media coverage is a great way to go.
If you were doing it in good faith, you wouldn't need to publish the actual code. Most likely you should publish an article and a flowchart explaining how the algorithm works. Publishing a partial chunk of code just creates a story that supporters who don't understand can parrot that "they opened their algorithm".
Exactly. Publishing what they have is the worst of both worlds - hopefully people will create flowcharts based off it, though, although it sounds like there will still be a low level of accuracy.
I still hear reverse-FUD about nvidia supposedly fully open-sourcing their Linux driver, when in reality they opened a tiny kernel portion of it that allows the main proprietary blob to connect to necessary kernel interfaces. You have to call out this bullshit when you see it.
Wait, what? AFAIU what you say is true, except for the part where the “main proprietary blob” does not run on the CPU. This isn’t as glorious as an actual open-source driver would be, but it does have meaningful advantages—e.g. you now have a ghost of a chance of implementing Nvidia GPU support on a non-Linux kernel, by uploading the GPU-side blob and rewriting the CPU-side shim as required. Or is the blob license-restricted from being used line that?
The "main proprietary blob" they're talking about is the userspace portion of the driver; the portion which does all of the heavy lifting. That definitely runs on your CPU. The only part they open-sourced is the kernel portion of the driver, which just exists to facilitate communication between the userspace driver and the hardware.
I say "why not both". Even if they are doing it only for good PR, we encourage it by giving them praise, because we should encourage things we want. (While remembering that they are not our friend, they are an entity we should pressure, and the way we pressure is by giving praise when they do things we like, and critcisim when they do not).
They should be commended for open sourcing something they don't understand because they fired all of the people whom understood it? Elon admitted as much.
Because the way he acts gives people every right to. I agree that he may be misrepresented, but if he is, then he has to shoulder at least some of the blame.
Not necessarily. What if the media company was bad for the health of democracy, and the billionaire's incompetence destroys the company's social standing and thus its ability to do more damage (even in the billionaire's own interests)?
Yeah, have to wonder how many people, if they had the money, would want to buy out Twitter just to wipe it out. Doesn't a huge chunk of HN hate Twitter and wish it were dead?
(Regardless I think that would be useless in the long run, since the millions of stranded users will still want another Twitter-like platform. And Twitter imploding without a designated archive will wipe out a tremendous amount of digital history.)
A lot of his decisions look pretty incompetent in the surface, like how could he not see how charging for verification devalue the system to whoever has the money?
Instead it could just be an intentional ploy to completely devalue Twitter disguised as incompetence. He can justify firing employees and charging for API access/verification as money-saving strategies, even if they're terrible strategies that have little chance of succeeding. And he could make enough people believe he's an idiot who makes things up as he goes rather than someone specifically driven or apathetic enough to run Twitter into the ground. Not to mention he was forced to buy them after changing his mind. Almost feels like a "so that's what happens" response.
I wonder how higher powers would be able to distinguish fake incompetence from real incompetence. Would they care how Twitter as a private company ends up if it's the case that it implodes from its own legitimately bad business decisions? It reminds me of how employers won't directly fire employees for discriminatory reasons, instead they make the employees' lives miserable so they're compelled to leave on their own, thus they escape scrutiny.
This is basically at the level of "9/11 was an inside job to bring down WTC 1, but WTC2 was destroyed in an unrelated but simultaneous terrorist attack"
> Yeah, have to wonder how many people, if they had the money, would want to buy out Twitter just to wipe it out. Doesn't a huge chunk of HN hate Twitter and wish it were dead?
> (Regardless I think that would be useless in the long run, since the millions of stranded users will still want another Twitter-like platform.
If there's not an obvious successor, right when its shutdown, a lot of those people might get their habit broken and find something better to do. I know Mastodon was held up as a successor, but it's unclear to me if that's actually capable of scaling to that level.
Mastodon is way too flawed to be anything but a niche tool for tech people and activists. I highly highly doubt such a system can cross the chasm. That doesn’t mean that’s a bad thing though.
Billionaires are billionaires not by literally storing cash. The rest of the society values their contributions and creations in the companies/corporations they run. Sure, they have some liquidity but the entire concept of resentment towards billionaires is essentially equal to resentment for the betterment of the world. There are some exceptions but for the most part, in a well oiled market, you can't just become a billionaire by fucking over people. See Adani and how it turns out for him: https://www.ft.com/content/5c0b6174-e66d-4fa5-89a5-6da1d69ab...
I encourage you to watch the C-SPAN recordings of the senate sessions where they brought in Twitter employees and journalists to cover what was in the Twitter files.
From your comment it sounds like you’ve been consuming the 30s soundbytes from those hearings and the misinformation spreading around the internet.
A long list of 3 letter agencies were compiling lists of citizens and journalists and sending them to social media companies to review for ToS violations.
There is a very real threat to civil rights here. When this cannon swings around and points back at LGBTQ, racial equality, stopping the war on drugs, etc. this is going to be “not pretty.”
And the hearings covering them were unbelievably shameful. Senators talk passed the guests in the room. Refused to abandon their “sick burn” scripts regardless of where the conversation went. Insulted their guests. Went in random directions of questioning that had little to do with the root problem…
At the core of this, 3 letter agencies (seemingly across the board) have decided that it’s acceptable to ask social media companies to prevent citizens from communicating on their platforms by selectively directing the attention of their moderation teams towards individuals. Whether this is legal, or a violation of 1st amendment rights, is for sure an open question.
Only one senator directly addressed that and only briefly by saying “maybe they’re trying their best” - a statement that doesn’t exempt anyone involved from following the law.
Is the government allowed to censor citizens by weaponizing their ToS for selective enforcement and, if the government can do that, where is the line drawn? How specific are they required to be? Can a platform ban all political speech and then only selectively enforce requests from the government without doing their own moderation? How far can we launder the 1st amendment through a public-private collaboration of enforcing ToS?
Honestly I’m not sure what the hearings were really meant for, the government is unlikely to hold itself accountable. At this point I do believe the ball is in the citizen’s court to bring suit against the agencies named in the Twitter files like we did with the presidential surveillance program.
The government requesting that the tos of a private company be upheld seems rather mild to me. Did we get the reasons for the requests in the released files? Were they trying to reduce foreign propaganda or public health misinformation or something else important?
Please don't break the HN guidelines like this. It's not what this site is for, and destroys what it is for.
What's worse, if you have a true point, then posting like this actually discredits the truth and gives people a reason to reject it. That isn't in your interest and in fact hurts everyone.
"Way too negatively"? We're talking about one of the world's most influential people who uses their power to randomly accuse innocent normal people of being pedophiles. There is no portrayal too negative.
It's healthy to have a normal amount of cynicism. They released it for a reason. "The goal of our open source endeavor is to provide full transparency to you, our users, about how our systems work."
Why be transparent (or try to appear transparent)? To convince people to trust your platform (or to recruit - which seems to be another goal of the post). Why would Twitter want or need to do this now? Well, there is a bit of context. This disclosure doesn't exist in a vacuum.
I agree, which is why I wonder what your motivation is to defend Twitter. You're posting about this for a reason. If I were a social media company, I'd probably have paid agitators to defend them.
If we are willing to not assume some borderline "it's what they want you to think" conspiracy play, obviously there was always going to be a lot of highly interested and qualified people taking a very close look at this and, at some point, there was always going to be very definitive conclusion of what's the deal with what they released.
If your play was "it's some source code, hence people will think we are open, and that should be really good for us", that would make you a very special kind of idiot in this space.
That was one of Elon’s core statements when he first talked about buying Twitter. If he had gotten it out sooner there would be an easier link between the two, but if you want more context just go read the old tweets and articles from the Twitter vs Elon days.
It's no secret that Twitter, like any other social media platform, is driven by user engagement and ad revenues. The more time we spend on the platform, the more valuable it becomes for them. With this new open-source algorithm, they're essentially crowdsourcing improvements to their system to better serve us the content we crave.
this move could be seen as a strategic PR play to boost their public image amidst the growing concerns around algorithmic bias and lack of transparency. By inviting the community to collaborate and address these issues, they're not only shifting some of the responsibility onto the users but also deflecting potential criticism.
Noone has mentioned this before - I don't know if it's really related, but afaik the European Union is thinking about requiring social media platforms to be more transparent when it comes to recommendations etc.
If you can already say "hey we have a lot already online!" then maybe the laws will become less strict.
> But the underlying policies and models are almost entirely missing... Without those, we can't evaluate the behavior and possible effects of "the algorithm
And neither can spammers find and test the cracks and edge cases that would allow them to break the system, that does sound reasonable to me. If they were public there would be an arms race between spammers/those wishing to game the system and Twitter engineers.
They’re explaining how it works without giving the specifics. Much like the US military explains how the nuclear deterrent works without disclosing detailed plans and control codes.
imagine thinking you need to read every file in a project to understand the architecture and which pieces are important for specific functionality you're looking to understand. Have you ever picked up a bugfix ticket for some code you didn't write?
It's fast to read stuff when you have the domain knowledge. The weights won't be a 5kb Scala file: they'd probably be a big binary file, which is easy to search it github/locally after cloning.
Otherwise, if they are provided, someone in the thread will surely point to them.
Princeton has a Code Reading 101 that all postdocs/professors must take, however in exchange for the Secrets of Speed Reading you must acknowledge every message with where you learnt those skills.
The context is relevant for indicating that they’ve familiar with the problem and have thought about these issues in depth. It’s also useful for not being accused of hiding their identity if someone thinks they have an unmentioned agenda. Argument from authority is bad when it’s of the form “I am an expert, therefore you shouldn’t question this claim”, not when it’s used to provide an identity to a previously-unknown name while also providing a cogent argument and supporting evidence.
It's disappointing the comments are so obsessed with the political angle to this that there's a total lack of appreciation (or discussion) of opening up the most influential social media platform in the world.
This is transparency theatre, not actual transparency.
There's no way to actually use this limited release to understand how or why any tweet is boosted, so we're in exactly the same boat we were in yesterday.
This sentiment has high correlation to driving conclusions from a very time limited information set. This isn't the only part that is going to be posted to github.
What is the net benefit from rushing to condemn something that can only be a net positive compared to the past alternatives? I don't understand the purpose of that approach. Help me.
The degree to which it is selective has yet to be determined.
Are you claiming total ignorance is superior to partial revelation? I think we would all do ourselves better to go live on a desert island and abandon everything about modern life. A shovel might be useful to bury our heads while we're there.
> Are you claiming total ignorance is superior to partial revelation?
I am claiming that this is at least sometimes true, yes. Not always, but sometimes.
You're the one claiming that partial revelation is always, without exception, superior to total ignorance. That seems unlikely. Propoganda is often partial revelation, are you saying it is always better to receive only propoganda than to receive no information at all?
I think there is objective value to understanding propaganda's origins and goals. Those who do not study history are doomed to repeat it. Those who do not understand propaganda are highly likely to be controlled by it.
Propaganda can be a pretty vague term by the way. Can you describe how the public coming to a better understanding of the inner workings of the worlds most influential social media site is merely propaganda?
> I think there is objective value to understanding propaganda's origins and goals
Of course, but this requires the propaganda be contextualized, which wasn't a part of the situation I was suggesting.
> Can you describe how the public coming to a better understanding of the inner workings of the worlds most influential social media site is merely propaganda?
You're begging the question.
Does Twitters disclosure of parts of the algorithm used (but certainly not all!) actually lead to a better understanding of the inner workings of Twitter? Or is such a release actually serving some purpose beyond transparency?
If we had that, I'd agree it would be good. But I'm not convinced we do.
Elon's done the exact same thing before at Twitter with his selective disclosure of material to friendly journalists in the "Twitter files". That, in my opinion, led to an overall worse understanding of Twitter's actions, not better.
Then the analogy fails to hold up to reality. There's plenty of information to contextualize. Assuming by default everyone but you is a naïve doe lost in a forest and therefore lacks the intellect to contextualize anything for themselves is undemocratic.
>But I'm not convinced we do.
That would likely be because, as mentioned previously there is still more to come out; it's not possible to be reasonably convinced yet. Going back to the original question:
>>What is the net benefit from rushing to condemn something that can only be a net positive compared to the past alternatives?
There's nothing, that's what the point of this thread/argument is. The portion that Twitter has opened (or even simply committed to opening) is not remotely enough to hold them accountable.
The code they've released here is less helpful than a single Helm chart.
>>>What is the net benefit from rushing to condemn something that can only be a net positive compared to the past alternatives?
Because this is useless, and worse yet, it's pointless and blatant virtue signalling. I stood up in defense of Musk's private bid for Twitter, but there's nothing worth licking his boot over here. The suits don't care. The Open Source community gains nothing. The users will never see, interact with or modify the recommendation code. Nobody will be able to meaningfully audit anything until Musk stops selectively burning the books at Twitter HQ.
If Elon wants Twitter, he has the money to go get it. If he wants my respect, he's got to do an awful lot more than making "the algorithm" public. This release is so pathetic that it's probably colored my opinion of Musk more than any of the opinion rags I've seen yet.
> Assuming by default everyone but you is a naïve doe lost in a forest and therefore lacks the intellect to contextualize anything for themselves is undemocratic.
Luckily not a claim I'm making. It's better for conversation if you reply to what I say, not misrepresentations thereof.
> Going back to the original question:
And I replied to it already, but I'll reiterate: I don't believe this change "can only" be a net positive, what makes you believe that is the case?
What is "one" and what is "zero"? Are you saying that information can't ever be misleading?
Like if I tell you that your boyfriend has been having secret meetings with some woman you don't know, with full knowledge that the secret meetings are because she's a photographer and he's planning to propose, have I improved things by disclosing the information to you in that manner? Were my actions a "net positive"?
I did find the article more enlightening than the source code. I always suspected the quality of a tweet did not factor in to its promotion, only its engagement, and now they have confirmed this to be the case. Now I understand better why twitter seems to be filled with people angrily retweeting what I consider to be low quality clickbait tweets.
As long as they don’t try to tackle tweet quality at all separately from engagement twitter will remain unappealing to me.
What does "quality of a tweet" mean, how might you measure it?
We pretty much knew this is how all social media works, becuase a) engagement is what they want, why wouldn't they be optimizing for it, and b) how else might you measure 'quality'? Back when this started, I have no trouble believing some well-intentioned engineers thought that engagement was a good proxy for quality. A bunch of users give it the "like", isn't that a collective assessment of quality? Who is to say what quality is, overruling the users in aggregate?
I agree it has the negative effects you mention; and I've read lots of people writing about this, it's of course not a new observation.
But I agree it's good to have an explanation of what's going on, even when parts of are what we basically knew was happening on all social media networks. confirmed is better than "basically knew", for understanding how these things that effect our experiences (and our society) work.
But why not shower Twitter with praise for doing more than any other social media company in a similar market position? In the best case scenario, we might inspire a transparency arms race. In the worst case, we merely signal that transparency of any kind is rewarded, and that's a good thing if we want more transparency.
> But why not shower Twitter with praise for doing more than any other social media company in a similar market position?
Because selective disclosure is often propaganda. If a third party had chosen what the release or verified this is what's actually running in production, I would praise them.
Considering Elon's incessant lying, self-promotion, and manipulation, it's impossible to be complimentary of this at all. Everything he does is in bad faith.
This is definitely incorrect. Weights for images, links, misspellings, etc are all laid out in the code and have been detailed by multiple people on Twitter already.
It would also be a total Elon move to confuse the open sourcing of Twitter's internal code with actual transparency.
You would need, at a minimum, a neutral third-party audit of Twitter's servers to conclude that the source code we see on GitHub is, in fact, the source code running Twitter. How often will they keep their GitHub repo in sync with their internal code, I wonder.
Presumably Twitter uses a version control system. But they scrubbed the history so that's also a point against their "transparency" claims. Without knowing the when and the why of changes you can't understand what you are looking at. People are pointing to that "author_is_elon" without knowing whether that was done before Elon bought Twitter or after.
But even then, git history can be faked.
> This is a great thing.
I disagree. It's the opposite. It provides the illusion of openness without the quality of openness, thus killing the debate once and for all.
I'm sure we can all think of examples where a power structure (a company, a country, a prison, a family) invited people in for a supervised tour that was less than honest in its presentation.
But really, if people respond to Twitter's actions politically, that response exists within a context that was certainly influenced by Twitter's prior actions.
"Opening up"? You must be kidding. Nothing is open there. It's just open-washing. A few nice diagrams, but how the services _actually_ work is still hidden.
I had to go to the second page to find this. I completely agree. I clicked this thread looking forward to seeing some intelligent discussion of the merits of the source and issues/interesting tidbits, instead I see a bunch of Elon Musk ranting and complaining and pointing out drive-by poor pull requests issues. I really wish the adults would start to talk.
I would definitely love a techy community that focused on technology like the hackernews of 10 years ago. If you know of one then please link me to it.
lobste.rs is pretty good, in terms of higher tech discussion signal to noise, I think, though it still does have a bit of a political bent. Email me if you want an invite.
One of the things that makes my spidey sense tingle is when people say oddly sycophantic things about Elon Musk. Twitter is big, it's important. It's not "the most influential social media platform in the world".
I only see one social media site posts being constantly reposted on global news organizations. Are all of the corporations, world governments, and leaders with tens of millions of followers actually wasting their time by dedicating their social media teams time to twitter instead of focusing on some other social media site thats more influential? Which one is it then?
It's quite odd to attribute objective analysis to sycophancy. I intentionally didn't mention him but here you are bringing him up and fulfilling my point. Who is the sycophant?
I would agree the quality of posts has gone downhill, and is way too political for my liking, but there's not much other place to go that isn't elitist.
I don't think Discord is a substitute for HN. If I know an exact community with a shared interest I want to join then yes, I can find their Discord an join it. But how am I supposed to come across that topic and become interested in the first place? HN (in its ideal form) is a place where I am introduced to a variety of random but potentially interesting technology-related topics, from programming languages to passion projects to lessons in business etc, some of which I will pursue and seek out a community for, and others which I only care about minimally, not enough to want to join a dedicated community for.
If you want to compare HN to Discord, I would say that HN is like joining 500 Discord servers that together comprise the type of content discussed and posted about on HN, and every day checking each one's announcements channel, and skimming through general chats. But you still have the issue of being introduced to new topics, for example if you're interested in the AI trends then your selection of Discord servers that comprise what's posted on HN would be dramatically changing in the past few months.
I've found Discord worse in every way. Almost every server I've been in is highly reactionary to any kind of accidental phrasing. Walking on eggshells is a constant process in any Discord server. It also descends into the same type of problems as any other chat services as it encourages short messages rather than long well-thought-out posting.
It has indeed felt a bit like r/LateStageCapitalism or r/antiwork in here, with so many low quality comments that all read exactly the same. I figured it was just because of the tech layoffs that that sort of sentiment has been gaining traction though.
It's actually shocking to see some of the discussion and understanding of the code here. I'd say the "part-time programmers" qualifier is even generous.
Great! But nothing is going to change until people realize that the problem is the feedback loop. It's not the recommendation engine itself, it's the fact that there's no way "out" of the feed that the engine produces. It recommends you stuff, you have little choice but to engage with it, and then it trains on that information.
This is the problem with most of social media today. It is a very well known problem in ML [1], but nobody is willing to do anything about it because it's a fundamental UX change. Facebook, Twitter, YouTube, TikTok, they have defined themselves by their recommendation engines.
reminds me of a story about a guy who was given a gift, a decorative plate with a rooster on it i think it was. didn’t care for it too much, but out of politeness put it on display on an empty cabinet he had. a while later someone noticed he had it and figured he liked it, so got him a similar decorative plate with a rooster on it. again, out of politeness, he put it next to the old one. now other people started to think he just really liked roosters, and started giving him little rooster statues and nicknacks. Eventually he just has a whole display cabinet of rooster themed gifts that he never really cared for to begin with, but people just assume he likes them because people keep giving them to him.
I had a friend who this happened to with ducks. You would go to her house and there were little duck icons and duck figurines and duck themed things everywhere.
When questioned, it turns out she never cared about or liked ducks. Someone gave her one and then it became the go to present for her for every occasion, for decades.
I am probably part of this problem - an enabler, so to say. I was hosted a nice lady with a house full of duck themed things (this was in southern England, probably not the same lol), and left her two (rather cute) duck-themed things when I moved assuming that she'd like those.
She was unequivocally happy when I did, but perhaps it was just because of the gesture.
(EDIT: also, the rooster/horse/duck example of feedback loop would make for a terrific blog post)
and in the end, he starts to like that decorative plates with roosters or at least thinks rooster on plate is a nice decoration if it is produced that much.
I think Instagram in particularly is bad in this regard. It seemingly becomes convinced that I care deeply about the subject of any post that I even momentarily linger on.
Don't you have a choice.. to not engage with it? If you didn't like it then assuming the metrics system is working correctly, this would be negative feedback to the ML model, causing said content to not be shown in the future.
1. Actively supplying negative feedback is sometimes hidden behind secondary menus, making it much higher friction compared to just...scrolling past. So most users don't spend the effort. Even with a dislike button, it's unclear what the system is learning. It can't know that I don't like this particular video because it's a conspiracy theory, and to stop showing me those. These platforms often don't even support explicit categories, so how would they know?
2. It's extremely high friction to teach the algorithm you're interested in something that it doesn't suggest to you! There's the whole unknown unknowns problem: how do you teach the algorithm you're interested in something that you've never seen before?
I still think Reddit has handled this the best. No system is perfect, but Reddit's challenges are much more manageable than the quagmire that TikTok, Facebook, and YouTube have gotten themselves into. I can just unsubscribe from r/conspiracy, and I'm out. Basically impossible to teach that to YouTube without weeks of careful curation. They think they're smart enough to know what I like, but they're not and never will be.
Eh, 1 is sort of why I see implicit negative feedback as more useful here. Namely, tracking the duration a user is probably giving their attention to a given item, weighted in accordance with how long you'd expect someone to give their attention to an item based on how long it is.
For example, I might see a some specific word or pattern of words in a tweet and quickly skip to the next one. That's very low friction but also a powerful signal. There are drawbacks (e.g. the long boring video that looks like something is about to happen but never does), but that's mitigated by combining this with other signals.
With 2, it's an explore/exploit trade-off. You try to explore as far and wide as you can while trying to avoid the things the user may dislike, all while sprinkling in just enough of the stuff that you know they'll like.
Yes, there are signals in human behavior that you can feed to your model. But no, it is never going to learn "on Mondays he works on his comics, so he'd prefer to see webcomic-related content" Don't sprinkle in your explore/exploit experiments. I know what I want, just let me decide based on what I'm in the mood for!
TikTok-style feeds are the absolute worst offender here, where they couldn't care less about what you think. They will serve you content, and you will either say "yes" or "no". So the only option for you as the user is to just wander through their content hyperspace. There's no structured way to jump between topics because everything lives in this formless content soup.
The other problem is that many social media platforms have you subscribe to the content streams of individuals directly. Individuals are high variance. How can you teach these engines that "I only care about this person's posts about pianos, not their terrarium hobby."
That's exactly the problem! I don't want to see them, but the algorithm refuses to ask me.
If they think that the length of time I spend on a post is 100% correlated with the amount that I want to see it, they really don't understand people at all. It's an embarrassingly shallow model of how how humans behave.
> It's an embarrassingly shallow model of how how humans behave.
It's a trade off. Viewing rates give back feedback 100% of the time. Asking users for a thumbs up or down gives feedback almost none of the time, and still might not be accurate.
Twitter could recreate a similar system: 1) auto-tag tweets with labels, rather than users needing to submit to subreddits 2) auto-sub most people some default set of labels 3) let them un-sub if they want.
Auto-tagging is not a business they want to be in, because they will invariably get it wrong sometimes and people will start drama over it. And they'd be right to: human communication is nuanced and you can't just "guess" what someone means within 280 characters.
People already use hashtags, what would happen if you could subscribe to them? It would create a whole new way of interacting with the platform, something closer to Reddit where people post to streams. But the fact that these streams can't be moderated (or even downvoted) might be a nonstarter.
Thank you! Working on a concept for a big org on what may become a large ML-based system one day. I knew about this feedback loop issue, but was too dumb to actually remember and face this problem. :) It's all over today's rec engines -- and yet, just like the things we're not shown in these systems, the problem itself seems to become invisible. Because it requires new thinking.
Reddit is a good example of an alternative system, and it works because subreddits serve as lower-variance composable streams as well as shared digital spaces. Even though your main feed is partially driven by a recommendation engine, it never contains any content that you didn't subscribe to (ignoring ads, of course). Instead, you may see a "posts from other subreddits" section that breaks up the feed explicitly to offer alternatives. If you're interested in a particular topic, you can drop into that subreddit, where you'll see pretty much the same view as everyone else. In all these cases, you're always in control.
I wouldn't be surprised if Reddit actually has the best data on their user's preferences, because they let them explicitly decide what they want rather than playing this "hot and cold" game like TikTok. Those platforms also only let you subscribe to individual accounts, so they have to infer what your interests actually are.
I feel like the Youtube one is good. You can mark videos and channels as "not interested" and Youtbe really knows me due to my account age and usage... It recommends me unknown videos and I tend to like them but also more mainstream stuff.
It doesn't matter how good they try to make their recommendation system, they will never know you like yourself. For example, when I go to the YouTube home page, there is a list of categories at the top that it's identified for me. I didn't choose these. I can't add or remove them myself if it's wrong. I just have to hope that I watch the "right videos" and it picks up on a new interest that I have.
But I already know what interests I have! I want to have videos about terrariums on my home page now, not in a week when I've watched enough. This is what I mean by recommendation systems not being good enough. They need to give the user more control over what they want to see, because they can never read my mind. Their recommendations will get even better with that information!
What did they learn though? They can't possibly know why I'm not interested in that tweet, because there's just not enough information. So all they'll do is stop showing me tweets from that one person. No way to express "yeah I'm interested in his tweets about music, not gardening." The problem is that mostly following people means that you can't teach the algorithm about your particular interests.
Topics are better, but there doesn't seem to be a way to make sure your tweet ends up in a particular topic. Again, you have to pray to The Algorithm that it will label tweets correctly. And if your particular interest isn't a Topic, you're out of luck.
Haha no, we just need to stop pretending as if ML can read our minds. It's a tool, it's good at recommending me content, but it will never know what I'm in the mood for on a particular Friday night.
That would be great (unweighting bluechecks) but they actually plan to go in the other direction: Starting April 15th non-bluechecks won't show up in the "For you" section (the algorithm timeline) at all. Unpaid users are being written completely out of the algo.
Inaccurate. Musk stated that people you follow will continue to show up. I only use the 'for you' feed when I'm bored and want stupid dopamine hits, I leave it on Following almost all the time. But that's on desktop, my understanding it that it keeps resetting itself for mobile users (of whom I am not one).
No. There's an implied willingness to listen to people in the first place when you're agile in response to feedback. You shouldn't have to bruise Elon's ego to get him to do not-stupid things.
This distinction you’re making relies entirely upon how you personally interpret the motives of the people involved. An interpretation that you’ve entirely invented for yourself.
Approximately every US news organization and Lebron James announced they weren't going to pay for checkmarks yesterday, so he announced a surprise twist where the top 10,000 famous people get free checkmarks.
…so he's now not charging people who can pay, while charging people less able to pay.
Though, the main problem is that Twitter is about as big in Japan as it is in the US, but Elon only thinks about US culture wars because his only friends are a few VCs and a retired postal worker crank named catturd2. So none of his new ideas are going to work there.
Criticizing somebody by speculating about their motives really only reflects badly upon you, especially if you’re going to accompany it with that much hyperbole. All you’re communicating is that you’ve replaced your ability to reason about a topic with some version of the fundamental attribution error.
> the main problem is that Twitter is about as big in Japan as it is in the US
It’s not even close. When you account for the value of ad impressions, the US market is worth about 3.5x its next biggest market (Japan), and it drops off very sharply after that. You can argue that Twitter’s policies should be less US-centric. But the reason they are that way in the first place is because the US market is their most valuable market by a long shot.
> Criticizing somebody by speculating about their motives really only reflects badly upon you, especially if you’re going to accompany it with that much hyperbole.
No, you can tell who he cares about because he replies to them, and because his emails were released in discovery from the earlier Twitter lawsuit and it turned out his friends put him up to it because they were mad the Babylon Bee got banned for a US culture war pronouns joke.
I certainly don't have to expect he'll make good decisions, since he has no experience with the business, is a completely atypical user, was forced to buy it in the first place, and has lost $20 billion so far by his own valuation. (May have lost a few millions more by firing that disabled Icelandic acquihire in violation of his contract and a few discrimination laws.)
The network effect is very very strong though. Even if someone makes a good competitor (Instagram is prototyping one apparently) I'd be surprised if bad decisions actually killed it.
> When you account for the value of ad impressions, the US market is worth about 3.5x its next biggest market (Japan), and it drops off very sharply after that.
I suggest reverse adjusting for how poor their ad targeting is. I've used it both places; the ads are actually good and relevant in Japan (…except for being for domestic apps, so not valid for tourists) but in the US they've always been nonsense. eg if you follow any doctors it will just assume you're also one and burn the budget of every medical ad it's got showing them to you.
What does that matter? If people find the content engaging then it will be amplified. If not, it shouldn't be there in the first place. This whole "AI / Bot swarm" excuse is just smoke and mirrors for "I want more people to pay twitter".
If you don't care whether the content is generated then you don't need to use a social network at all. GPT can generate you an endless interesting feed no problem.
I believe LeBron James said recently he isn't going to waste his money on a blue checkmark, so it should be interesting to see what stays and what goes.
Most of the major news outlets are not doing so either. The Elon stans are crowing that this will be the long-overdue end of legacy media, but it strikes me that the new 'blue check twitter' might end up becoming even more of a social bubble than what it replaced. There are so many low quality accounts sporting a checkmark now that users who value substance will soon be incentivized to just block anyone they find annoying.
Good find! It'll be funny to see if the incumbents respond with 'don't do me any favors.' Also to see whether Musk's frens sulk aboit him selling out to the elites or so - their gratitude has an extremely short half-life.
Seems dumb of them. Cost is trivial and their competition that isn’t so politically motivated will have a much further reach.
The smart move would be silent on the policy change, pay, and support rival platforms as they can. Instead they will eventually pay and look like they lost.
> Seems dumb of them. Cost is trivial and their competition that isn’t so politically motivated will have a much further reach.
It's wild hubris for twitter to try to invoice/penalize the very users and organizations that make twitter anything but insolvent. There should be money exchanged here, but it should be flowing generously and most importantly in the other direction.
For the NYT to verify their official accounts plus those of their reporters (using the Twitter Blue Affiliations feature) would be $1m annually. This, for a budget line item that has heretofore been $0. In this economy, that's a reach.
LOL they are desperate for reach. Incredibly so; have you not listen to any podcast by them? They are begging people to go to their site. They get a fraction of the organic traffic they used to and nearly everything is driven from other site like Twitter, Google News, Facebook, etc. The internet age has not been kind to classic news orgs.
correctly writing words and punctuation on the page digitally printed by "the legacy news company that's doing the best online of anyone" doesn't matter at all? isn't that like the bare minimum of what their job consists of?
I was responding to a post saying they weren't getting online traffic by saying they are getting online traffic. Nothing about the quality of their content.
An affiliated account to a verified org is $50 per month per seat, so NYT would have to authenticate 1,647 affiliated accounts to reach 1 million dollars per year
Are all of these 1647 reporters (they have that many??) and posting on Twitter? That’s a lot of traffic generators or not.
Surely they could just do the bulk with 100 or so.
You're suggesting the NYT further tier its reporting ranks, along with all the internal difficulty that would entail. For ex: obviously the 100 have to include the most senior reporters, who are also older and therefore the least likely to create the viral content NYT wants affiliated with their account, so immediately they probably need to look at a much larger number. For another example: social media is different for each reader, or from the other side, each reporter has a constituency. In one season, the fashion reporters are driving views, while the following season it's the European war correspondents or the economics reporters (and all of these desks have subdivisions that wax and wane in popularity).
And all that discussion so that they can spend $72k annually with Twitter, a y/o/y increase of $72k from last year. With no guarantees of reach, because the whole paid-only verification thing is an experiment that began an hour ago. Let me just say that this whole pitch is going to be...difficult... at the point in the economic cycle where we find ourselves.
The NYT has revenues of 2.1 billion. I’m sure they have a marketing budget and probably already spending money on Twitter to get traffic. This isn’t something strange.
Facebook did they same thing btw, just more gradual. For years they changed the algorithm slowly to take away reach from Pages only to offer it back as long as you paid.
Parent didn’t say it’s not “political”. It’s reasonable for a wealthy person to feel that a system that discriminates against the poor is not a system they want to participate in.
(Note that I use discriminate in the literal sense, as a simple statement of fact.)
But the example you give is an appeal to a universal moral good. Not partisan politics. So despite saying it’s not not political, your justification is that it’s not political.
Also, how did you get a blue check before being able to buy one?
Some people unfortunately view concern for the poor as political. However my point of mentioning politics is to say that “it’s political” is not any kind of gotcha when it was never denied as being political. Regardless of the actual justification being political or not, the “political” gotcha is nonsense.
That’s bullshit. Virtually everyone agrees poverty is a problem. Sure, the welfare state feeds the cycle of poverty, but it’s not like that was the goal.
Absolutely not bullshit. Some cynical people on the internet believe this, and that's what I thought the person I was replying to was saying. It is an extremely low bar to say "some people believe X", and I don't know why you care to question that. Even with your own reply you say "virtually all people agree" and your use of "virtually" acknowledges that not everyone agrees with you, therefore some people do believe what I say. This is anyway such a silly tangent and was not even my point.
Like it or not but it's the twitter that gets value from celebrities. How many people are on social networks jusy so see what their fav celebrites are doing?
The problem for twitter is it isn't the only game in town when it comes to social media, not by a long shot. They're not even in the top ten. They're a megaphone in a large pile of megaphones, and those other megaphones don't bite the hand that picks them up.
It sets the default to zero, but apparently can range up to 100. So... what modifies it? (The answer is probably in there somewhere, but I'm sure someone will find it before I do.)
I think it makes sense for out-of-network. However I see no value for boosting among people that I have followed. If I have followed them they definitely aren't spammers (from my PoV).
The trouble with spammy jokes like this is it discourages companies from bothering with open-source in the future. I know I'd be less likely to champion an initiative like this if I thought it might blow up in my face.
The irony is that I prefer Mastodon's sort by time and don't try to be clever approach to this expensive and futile attempt to feed me an endless stream of click bait. I objectively spend more time on Mastodon than on Twitter at this point. It's more engaging for me. It's how Twitter used to work when it was still nice to use.
If Twitter wants to put a stop to the user exodus and save lots of money in the process, here's what they could do:
1) Add an off switch to the for you feed. I'll click it right away and never turn it on again. Stop wasting minutes of CPU time on my behalf. I never asked for it. It doesn't do anything for me that I need or want.
2) Sort by time, filter by hashtag. Twitter used to be about real time information. I don't care about things that happened days or weeks ago. I don't need to see all of it. This is the core feature that made Twitter popular. Mastodon has it and it is absorbing users from Twitter by the millions. It still works. Restore this feature and make it the default.
3) Join the fediverse. That's where a lot of the former hard core users went. They still exist. They still post messages. They still engage with each other. They just don't use Twitter anymore. Allow people to follow mastodon users. Allow mastodon users to follow Twitter users. Not that hard to implement and probably would do wonders for user engagement.
But it's not sticky and it's easy to miss when it forces you back to For You. Less of a problem on desktop because you can write a trivial extension to force Following, but not an option on mobile, unfortunately.
I think, very recently, it was made sticky again on twitter.com, yes. I'm not sure when since I use an extension anyway, but it definitely wasn't sticky about a month or so ago.
As near as I can tell, Mastodon doesn't really have #2 in the list above. Last I heard, the social architecture was hostile to comprehensive indexing of the entire fediverse for search.
That's probably one of the biggest reasons that I have remained on Twitter even after setting up a Mastodon persona.
// we only keep unfollows in the past 90 days due to the huge size of this dataset,
// and to prevent permanent "shadow-banning" in the event of accidental unfollows.
// we treat unfollows as less critical than above 4 negative signals, since it deals more with
// interest than health typically, which might change over time.
val unfollows: SCollection[InteractionGraphRawInput] =
GraphUtil
.getSocialGraphFeatures(
readSnapshot(SocialgraphUnfollowsScalaDataset, sc),
FeatureName.NumUnfollows,
endTs)
.filter(_.age < 90)
/**
* These author ID lists are used purely for metrics collection. We track how often we are
* serving Tweets from these authors and how often their tweets are being impressed by users.
* This helps us validate in our A/B experimentation platform that we do not ship changes
* that negatively impacts one group over others.
*/
... Metrics tracked in AB test. So even if it's not explicitly encoded in the algo (or implicitly through some of the features plugged in), they'll pick the winning cell as long as it doesn't hurt Elon's metrics (I'm just parroting the comment you quoted).
It doesn't have to be in the algorithm for the systems to be tweaked to please Elon vanity metrics.
[I've been running lots of ML AB tests over the years, some in organizations of similar size & complexity as Twitter]
That lines up with reporting from Casey Newton a few days ago where a handful of VIPs e.g. Musk, LeBron James, AOC were being used as weather vanes to understand what the algorithm was doing.
It definitely isn't just metrics. Any algorithm change that negatively affected Musk was clearly not going live.
> I suspect that there was another name there before
Who ? Musk is unique in being obsessed with being liked and relevant.
All of the other social CEOs including Porag and Jake have never really cared that much. And none of them participated in contributing content anything close to what Musk does.
Why is Musk obsessed with being liked? There was a hoax last month about the algorithm being tweaked to make everyone view Elon’s tweets which was provably false.
It is very much likely to be the CEO a company wanting to understand his companies product.
> a hoax last month about the algorithm being tweaked to make everyone view Elon’s tweets
Didn't this just reveal that they're A/B testing on Musk's tweet performance? At the very least they're avoiding regressions, and I guess any incidental improvements to it won't be reversed, so isn't it fundamentally the same? Unless we're taking the word "everyone" literally, I guess.
I don’t remember the specifics but I think there was a case where they accidentally amplified his tweets too much and they did reverse that change. He tweeted about it but I think it was a couple months back so it would take a bit to find.
Yes tabloids, both known for making up stories and sensationalism, all sourced from the same single blog article, easily confirmable looking as false at the view count for Elon’s tweets, explained by engineers as exactly what we saw here (logging output not changing output) and I don’t believe you re: seeing his tweets despite blocking him. Nobody else had that happen.
We can't comment on why, but there's no rational way to watch his behavior and assume he isn't obsessed with being liked. He constantly tweets about his own tweets performance, makes humiliating appearances on stages, and pretty much terrified that someone might do something 1/100000th as awful to him as he and his family have done to others.
> He constantly tweets about his own tweets performance, makes humiliating appearances on stages, and pretty much terrified that someone might do something 1/100000th as awful to him as he and his family have done to others.
Ah so he makes jokes, and is well known enough to need security precautions. This makes him “a whiny baby”. Solid argument.
The guy is literally creating the next gen intercontinental ballistic missile system and military delivery for the US, but according to HN it’s whiny for him to need security. Right.
Oh and I forgot, he also helps ukranians vs the Russians. Not a good idea if you want a risk free life.
Someone wants to measure their business after paying what was a reasonable figure when they offered it, but that became overpriced once the market changed.
Sounds like fairly rational behaviour.
HN isn’t a site for making unfounded personal attacks on others.
His offer included a meme number and was overvalued even in April '22 (by 54% !), claimed he was going to combat the spam bots, and then tried to back out, citing the same spam bots. The event was described as a "hostile takeover" at the time [0], a term which has stuck all the way through to most summaries of the event [1]
A hostile takeover has a specific meaning. It's simply a takeover attempt where a buyer approaches shareholders instead of the company's management, this is usually (if not always) because the latter fails. And it always comes with a premium and drama, precisely because it goes against the wishes of the management of a company. And the premium is also logical because presumably shareholders of a company "believe" in that company, and an outsider now wants to take it over. A good example [1] is when Inbev (massive multinational beer company) purchased Anheuser-Busch (Budweiser). It made the purchase of Twitter look downright cordial, but was probably outside of most of our bubbles.
Yea, to make people buy your company you need to offer a premium. He backed out saying he was lied to about the current level of fake accounts, which is also reasonable if that was the case.
It would have been reasonable if he had performed due diligence and came to that conclusion, but he explicitly chose to waive due diligence, and then tried to renege anyway.
As a person who occasionally uses Twitter, but has no interest in having an account, he has dramatically improved the experience. Before he took over it was made impossible to view replies from a user, users were spammed with popup login dialogues that could not be closed with a single click, and more. I expect Twitter probably drove more people away from the site with all this nonsense, than they coerced into registering.
I also think many of the changes to make Twitter more open, such as now open sourcing the recommendation algorithm, publishing view counts (which indirectly makes it clear when something or somebody has been shadow banned), and more are all big steps forward in creating a much better and open service for all. Really most of every change he's pursued since taking over Twitter, from my perspective at least, seems to have been big steps forward.
He accused Twitter of being politically biased to the left and suppressing tweets, so he bought it to "allow free speech", except in his mind free speech apparently equates to being able to promote right wing propaganda.
As a platform, sure it can be better, and it wasn't exactly under the best management before (although the Twitter file leak actually made the old management look better than everyone thought). But lets not pretend that Musk is in this for the platform - he wants control of information to boost Republican leaders because in his mind, he is the savior of human race, so the best leadership is that which allows him to do the things he wants to do.
> He accused Twitter of being politically biased to the left and suppressing tweets, so he bought it to "allow free speech", except in his mind free speech apparently equates to being able to promote right wing propaganda.
I'm not a fan of Elon at all at this point. That said, this is more an indictment of the tech industry than Twitter alone. Most employees are some form of left, many of those employees sit on committees, have outsized wealth compared to the average citizen, donate accordingly, the executives among them have power, etc and tech plays an outsized roll on our lives now. This was made a lot more evident when Twitters head of trust and safety was interviewed and she was unable to identify, or had let situations go on, that were offensive to people you'd commonly call right where anything that was offensively to what people would commonly call left was getting combed over and at times over enforced. I'd also say that my perception of her was more that she had large gaps rather than outright hate. The biggest one that comes to mind is "#learntocode" which a bunch of journalists not only wrote articles but also used that hash tag on people working blue collar jobs, such as coal mining, that were under threat at the time. It's an isolated example, but that hash tag stayed promoted for quite a while.
This isn't to say that right-leaning oppression and left-leaninf oppression at categorically the same or have the same categorical effects. It does point to that we have larger class issues at play.
Sure, except Twitter doesn't have free speech. Twitter can and does issues bans for thing said, notably in the Kanye case, and when Elon banned the elonjet account.
It's either free speech absolutionism, or not free speech. If you claim you are for free speech in the moral sense of the word, you have to let everything be posted.
As soon as you start removing things, it becomes your version of controlled narrative. And there is nothing wrong with just that, because of course you want to minimize potential damage. But then you have to be responsible for the things you let slide.
So when Twitter allows right wing rhetoric, a.k.a conspiracy theories about Nancy pelocies husband, pushed by the CEO, or things like Fauci gain of function research, it's clearly a controlled narrative.
I absolutely will criticize Elon for his behavior around the twitter buyout. He's handled layoff/firing decisions extremely poorly. He's alienated a large portion of the ad buyers. These are not unfounded attacks.
Elon does not need you going around personally trying to shut down other people's discussion. That's cult like behavior.
It was going to be tough to cut Twitter regardless of who did it. Advertisers were pulling out before the deal was even closed, mainly because of Tiktok.
How are the people that disagree with you in this thread trying to shut down your discussion?
If the “normal way” of doing things worked then Twitter wouldn’t have been in such a bad state when he took over in the first place. An unconventional approach brings more risk but can also end up being worth it.
Would I do things differently if it were up to me? Sure, but it isn’t, and I can at least appreciate that they are trying to move quickly & try different things. I’m reserving my judgement for whether or not this approach works in the long run.
Twitter wasn't in that bad of state. If anything, it just needed tweaking to unlock the value it should have had given it's place in the media. Now, Twitter is in a terrible state financially given the debt Musk loaded onto it for the purchase.
The problem is Musk is learning the hard way when taking something over is that the old crew is never as dumb as he hoped they were, and he wasn't as smart as he thought he was.
A company that is 16 years old, still not profitable, & with no concrete plans to achieve profitability is totally dysfunctional and probably wouldn’t survive a high interest rate + likely recessionary environment anyways.
> If the “normal way” of doing things worked then Twitter wouldn’t have been in such a bad state when he took over in the first place.
The normal ways of measuring the business, namely EBIDITA, DAU, MAU .., worked. They reliably reported the state of the business. Elon Musk was the only person in the financial market who looked at those measures and interpreted them as meaning “add US$1B annual debt service.”
That is "measuring their business" only if one of the main goals for their business is for it to promote them; because this is exactly what they measure here.
It’s the CEO being able to write something and see where it ends up after going through the pipeline. It does not increase views. I can repeat this, other can repeat this, or you can look at the code.
> which of these groups do you think that they use as a control?
When you run an A/B test you randomly divide your users into groups, one (treatment) getting the new behavior and one (control) getting the current production behavior. So your question doesn't make much sense?
Depends on the question. If you want to answer a question like “does change X increase engagement?” then a straight A/B test works. But if you want to answer one like “does change X increase engagement while (favoring/not favoring) group 1 over group 2?”, then an A/B test plus measuring groups 1 and 2 will not work without a control group, because without controls you don’t know if any changes to engagement for your measured groups are significant. There is some threshold of change to the engagement for the measured groups which is too small to be significant, and you should ignore results that only measure that noise.
I think a lot of it is lies... I used to observe that tweeting anything remotely political went straight to trending timelines, so did tweets about crypto and NFTs...
I doubt that Twitter and most of these social platforms are driven by Ai... Maintaining and changing complex algos could not be done on a rapid pace like what occurs now... I think Twitter has moderators, and scripts that control everything, and it can be more easily adjusted to tweak what is visible on the platform based on whatever agenda they want to represent (politics, revenue, PR/damage control).
Twitter frequently bends the rules to serve celebrities, politics, and sponsors and for that they need to be able to quickly adjust scripts. True Ai is meant to function on it's own with minimal intervention, and therein a simple change to a massive logic scheme would completely FUBAR everything.
I think there are rooms full of people that filter and either promote or suppress posts on Twitter every day, namely suppressing any tweets critical of the platform and it's owner... I've noticed that at night time hours, moderation of Tweets is less restrictive as a cue to what goes on there.
There are certain topics that are not moderated as heavily as others (3d Design and other non-controversial topics), and some topics (for example music and porn) that are restricted heavily visibility-wise because they force people to run ads to rank in trending (because it generates a lot of opportunistic money for Twitter as a platform).
Users that are critical of the platform and Elon and his allies (for example) can easily be neutered by moderators and put on shadow ban for any period of time in order to preserve the illusion of calm concerning Twitter operations. Complaints about Twitter only become visible when the majority of the audience tweets about problems (as that can't be moderated out without exposing moderation).
It's pretty much all smoke and mirrors there in order to maintain order in my opinion, and it's pretty much futile and torturous to people who just want to create and seize opportunity for their business without spending tons of money on platform marketing...
There is absolutely no reason to believe there was another Single user getting this treatment before. The Elon-case was just copy & pasted as an ego-stroking hack.
You would think someone forcing themselves up to the top of a feed designed to catch your attention would be a little bit more memorable than “I had to look up what he was even called”. You didn’t even know what his title was, ffs.
I’m sure your simping for Elon is highly appreciated. Maybe he’ll let you taste his boots next.
The code under discussion has nothing to do with forcing anyone or anything to the top of a feed. It’s entire observable effect is to _measure changes in engagement_. Any use beyond that is speculative.
However, if you believe that it was to be used as part of a system for increasing engagement, then you are asserting that it is a system with no control groups, which is a stupid mistake that wouldn’t be made by an undergraduate taking their first statistics class. They wouldn’t make that mistake even on the first day of that class, because they took a class in statistics in high school!
We all know said code does exist and just wasn’t part of the released source. It was very clearly used, although I know it goes against the cult so you yourself can never dare utter those words.
But no, please, go off on your tangent that is not at all centered in reality.
Such a system, which certainly exists, was also made extremely quickly. We all know the timeline. We all have seen the rants and then the immediate effect of manual changes made to please the rants. I could easily see control groups going out the window when your boss expects their tweets to be prioritized yesterday.
I also like how confident you are in a company’s ability after they explicitly laid off and fired the majority of its staff. Generally speaking, when that happens, people lose confidence rather than gain it, but here you are proving us all of wrong. Gold star for you, maybe you’ll even get the other boot now!
I’m sure it feels so rewarding to simp for a man who will never even look in your general direction, let alone talk to you or know all the great things you said about him and his companies.
Yea, I’m not willing to engage in any conversation with someone who uses ad hominem attacks. I should have noticed your previous one; I can only conclude that I wasn’t paying enough attention.
No. Not a chance. Elon’s claims of mismanagement at Twitter have merit, though not nearly as much as he think, but the one thing that’s undeniable is that the previous “owner” (well, the CEO), is an adult that didn’t care about this shit.
It is usually better for major criminals to confess and turn themselves in too. "I bet" doesn't read "it is morally better." And why jump to compare publicly vs privately doing it instead of doing it vs not doing it?
Why, exactly, should the CEO of Twitter care about their own tweets getting maximum attention? If anything, this takes engineering effort away from customers because special implementations like "is_elon" have to be added.
Isn't it natural for the CEO wonder about his own tweets, and use those as a proxy for the function of the platform as a whole--esp as relates to "VIP" tweeters who Twitter arguably wants to keep happy. In fairness to Elon, his repeated inquiries about his own tweeted shined light on several legitimate bugs in their algorithm affecting VIPs.
No, it isn’t a problem if the a CEO of Twitter doesn’t care about their tweets. The proper task of any CEO is maintaining the company as an immortal entity.
If the general public cares about the CEO’s tweets, then necessarily there will be a danger that the death of the CEO will ignite a crisis in the general public; e.g. Steve Jobs at Apple.
I never suggested the CEO needs to be the star tweeter. What I am saying is that the CEO needs to be a tweeter, i.e. personally invested in his/her company's product. Companies with CEOs who don't give a shit about about their product/users tend not last long as "immortal entities".
So many unnecessarily cynical takes here. Let's say you were in charge of a large legacy system that some segment of customers complain about it not working for them as well as other segments. How would you know whether their complaints are valid unless you measured it? You have to know first. So measure it.
Yeah, but then what do you do after you measure it? Nothing? No, you make decisions differently so as not to offend whoever is part of the criteria. For example, can we agree that we don't want an "author_is_flat_earther" flag? Because who gives a shit if Twitter makes a change to their recommendation engine that negatively affects flag earthers? Just because something is only used for A/B testing doesn't make it completely inert.
You can dismiss the complaint without measurement if you are confident in two things:
1. Your system does nothing to actually segment this specific group by their identity.
2. You are confident that the systems you have set up to reward good behavior and punish bad behavior are accurate.
If both of those are true, you know that even if the group is being disproportionately negatively impacted by some form of recommendation/moderation, that it is only because that group disproportionately participates in behavior that is bad for the platform. That isn't a problem. It would actually be worse for the platform overall if you did anything to appease that group.
> ...it is only because that group disproportionately participates in behavior that is bad for the platform. That isn't a problem.
That is exactly what Twitter's stance has been all along (in the pre-Elon era) and it IS a problem for the product because people being silenced due to their own bad behavior (example: misgendering transgender people) feel an injustice is being done. The rule-makers get to set the range of acceptable discourse on Twitter and those to the right of center have felt unfairly disadvantaged by the way it was done in the past.
Over time this has eroded trust in the product. Just because people aren't being labeled and ranked based on whether they are red team or blue team, the people deciding what "good" and "bad" behavior looks like on the platform have the power to disproportionately impact these groups.
Data isn't going to tell Twitter whether to allow or disallow misgendering people. You either think that is bad behavior that shouldn't be allowed or you don't. Disallowing it is not disadvantaging Republicans. It is stopping behavior Twitter has deemed is bad for the platform. As I said in point 2 above, either Twitter is confident in those decisions or not. Data is worthless when it comes to a moral decision like that.
And if we accept hat Twitter believes (or more accurately did believe) that misgendering people is wrong, who cares whether people who want to do it feel an injustice is being done? Would anyone say that deleting spam is an injustice to spammers? You break the rules and you get punished.
> Data isn't going to tell Twitter whether to allow or disallow misgendering people. You either think that is bad behavior that shouldn't be allowed or you don't. Disallowing it is not disadvantaging Republicans.
If one of the defining characteristics of a political/religious/cultural group is having a particular ethical view, then enforcing a contrary ethical view against them is disadvantaging them and discriminating against them. Now, it may in some cases be morally and/or legally permissible, or even justifiable, discrimination, but it still is discrimination, and it is still disadvantaging them.
> Would anyone say that deleting spam is an injustice to spammers? You break the rules and you get punished.
Worldwide, many jurisdictions have laws against discrimination on the basis of religion; although it is less common, some jurisdictions also have laws against discrimination on the basis of political belief. A law prohibiting discrimination on some ground, is evidence that some people believe discrimination on that ground to be immoral. By contrast, I've never heard anyone suggest that spammers should constitute a "protected class", and I'm not aware of any jurisdiction which treats them as one.
Some people believe that there is nothing morally wrong with discrimination on the basis of religion and/or politics. Other people think there is something morally wrong with it, but if there is a conflict between the right to be free from religious and/or political discrimination, and the rights of LGBT people, the rights of the latter morally ought to take priority. Spam is irrelevant to that ethical debate.
>If one of the defining characteristics of a political/religious/cultural group is having a particular ethical view, then enforcing a contrary ethical view against them is disadvantaging them and discriminating against them.
I don't think misgendering people is a "defining characteristic" of Republicans and if that is, the Republican Party is in a pretty sad state considering all the bigger problems in the world. And if that qualifies as a "defining characteristic", there are plenty of other counter examples of society accepting discrimination as you define it. Banning polygamy would be discriminatory against Mormons is one. You could even argue that a full abortion ban is discriminatory against Jewish people.
>some jurisdictions also have laws against discrimination on the basis of political belief.
Notably not in the US where Twitter is based and were most of these complaints originate.
> I don't think misgendering people is a "defining characteristic" of Republicans and if that is, the Republican Party is in a pretty sad state considering all the bigger problems in the world.
Some religious conservatives are convinced that referring to a transgender person by their preferred pronoun is a sin, even a serious one, for which they will be judged by God. Even religious conservatives who don't personally subscribe to that viewpoint, see it as one they are morally obliged to respect and defend. [0] While that doesn't describe all Republicans, obviously there is a significant overlap between Republicans and religious conservatives. And for a devout religious person, their religious beliefs are one of their defining characteristics–they are a huge part of their life, even their very identity, who they understand themselves to be.
Even those conservatives who think it is okay to use a person's preferred pronouns, will adopt a much more restricted stance on the topic than many trans activists. Many will insist on it must be voluntary rather than mandatory, and defend the freedom of conscience of those who take a more conservative stance than they do – which is an expression of both religious and political beliefs about respect for individual freedom and conscience.
Some conservatives are willing to use a friend/colleague/acquaintance's preferred pronouns when interacting with them, but will refuse to do the same for a criminal in the news. Look at Wikipedia to see people who vehemently insist that you must use the preferred pronouns of a dead school shooter or executed murderer – and even if the family of the victims publicly objected to it, that wouldn't change their mind.
> Banning polygamy would be discriminatory against Mormons is one. You could even argue that a full abortion ban is discriminatory against Jewish people.
Prohibitions on discrimination are never absolute, they always permit exceptions – so the existence of exceptions is not an argument against the existence of the prohibition. And whatever the merits of those specific examples, they are actions, not speech. Society traditionally gives religious minorities far greater latitude with respect to their beliefs about what they can and can't say, than their beliefs about actions which aren't predominantly expressive in character. Jehovah's Witnesses who believe it is a sin to salute flags or recite pledges of allegiance, Quakers who believe it is a sin to swear oaths, etc.
> >some jurisdictions also have laws against discrimination on the basis of political belief.
> Notably not in the US where Twitter is based and were most of these complaints originate.
While the US currently lacks federal laws banning political discrimination, state and local laws sometimes do ban it, see [1]. Some of those laws are specific to certain contexts (e.g. housing or employment), and so may not be applicable to a social media platform such as Twitter. However, given increasing concern among conservatives about political discrimination, it seems rather likely that we'll see more state laws enacted on that topic in the future.
> Some religious conservatives are convinced that referring to a transgender person by their preferred pronoun is a sin, even a serious one, for which they will be judged by God.
I don't expect their empathy. I expect them to experience consequences for their actions. The consequences can stop when the actions stop and aren't expected to be repeated. I expect some subset of them to have an understanding of cause and effect, and thus learn. (If the lesson they learn is "change actions" rather than "change mindset producing actions", that'll do.) I expect the rest to complain about experiencing consequences for their actions.
> I don't expect their empathy. I expect them to experience consequences for their actions. The consequences can stop when the actions stop and aren't expected to be repeated.
Or, they can turn around and try to impose negative consequences on the people who are trying to impose negative consequences on them. Which, one might argue, is exactly what is happening in several state legislatures in the US right now. And then the fight goes on until one side wins, or there is some sort of "peace deal".
That's what happens with irreconcilable values differences, yes.
But I wouldn't frame it as cause-and-effect like that: one side doesn't attack the other as a response to experiencing consequences; they attack the other pervasively at every opportunity, and sometimes experience consequences for doing so.
If one side believes they are completely innocent and the other side are just plain evil – and the other side believes the same things right back – isn't that how civil wars start?
We have a democracy precisely to avoid such things. I'd prefer the strategy of "decide to actually start winning at every opportunity"; it would be a novel change in strategy.
Let's stop treating this as "maybe there's a way to convince people", understand that there is no way to convince some people, and instead just win. Win, and keep winning, and use those wins to eliminate things like voter suppression and gerrymandering, and then never lose again.
Try as hard as you might, there's no guarantee you can win. What happens to "win, and keep winning... and then never lose again", if you never "win" the first time? The US political system is rigged (arguably by design) to favour conservatives. To successfully undo that rigging requires not just winning narrowly, it requires winning decisively. But how do you win decisively when the system is rigged against you? It might be about to get even more rigged–if the upcoming SCOTUS decision in Moore v Harper embraces the "independent state legislature theory", all efforts to prevent gerrymandering by state legislatures would be dead in the water. So, what if you don't win, what if you lose–what then do you do?
And even if your wildest dreams come true–if you win too big, the other side may turn around and say "democracy isn't working for us any more". If it gets to the point that a significant minority of the population (say 20-40%) no longer believes that democracy is in their best interests, democracy's days are numbered. Especially if that significant minority has a great deal of wealth, influence and power. It could end in the peaceful negotiation of a "national divorce"–and there are many worst ways it could end than that.
If it were easy it would be done already. But the important strategy is to make sure it only takes winning decisively once, rather than winning decisively and leaving room for getting undermined in the future. Priority #1 with the next majority should be eliminating voter suppression, eliminating gerrymandering, supporting universal vote-by-mail, and all other factors that prevent the outcome of democracy from actually reflecting what the majority of people want.
> the other side may turn around and say "democracy isn't working for us any more".
They do that already, whether it's true or not. That's not a reason to decide to lose.
How do you compromise with people who use that as a way to take advantage of you and refuse compromise in return though? The result of 'let's live with them and treat their actions as good faith' has consistently been 'we get screwed'. What do you propose?
Am I misunderstanding? Are you suggesting that penalising those who misgender transgender people is unfairly disadvantaging people whose political views are right of centre?
It's not about what I personally believe or think is unfair. It's about what Republicans (broadly speaking) believe. There is massive resentment from people on the right who think the Twitter rules unfairly elevated some political opinions as good/correct/acceptable while treating others as unacceptable.
The handling of trans issues is just one example to illustrate the problem here. People on the left think trans rights are human rights while people on the right think a lot of trans issues should be open for discussion, legislation, persecution, etc. I think if we're being intellectually honest most would acknowledge that as a country we are far from consensus on many of the details here (bathrooms, girls sports, etc), and yet Twitter's rules and enforcement actions behaved as if the leftist view of transgender people is the only valid and permissible view.
The handling of January 6 and the banning of Trump is another example.
These things are Rorschach tests; people apply their biases and reach very different conclusions about what should be done. I don't claim to know the solution, I'm just trying to sketch out the problem with the way things were creating a climate where a big segment of the US felt unwelcome and resentful toward the platform.
This presents a problem for the platform, since you can't afford to alienate large double digit percents of the population if your mandate from shareholders is to grow mDAU by any means necessary. In that context, having some metrics tracking in place to measure the impact of algorithm changes on democrats and republicans to see whether impact is disproportionate is a completely rational thing to do.
While I can understand there may be debate around various trans issues purposefully calling someone something when they’ve politely asked you to call them something else isn’t up for much debate. Seems like common courtesy/politeness.
I see your point about losing users potentially but I would argue that Twitter’s intense focus on the US (as shown by the democrat/republican metrics) and trying to placate everyone is actually a negative for their business. There’s billions of other internet users outside the US. Shifting focus to serve them instead of focussing intensely on trying to please both sides in the US (and failing) would probably deliver better value for their shareholders.
> purposefully calling someone something when they’ve politely asked you to call them something else isn’t up for much debate. Seems like common courtesy/politeness.
It can be if the thing they're asking for is perceived to be untrue and the person being asked is a big stickler for that sort of thing. If you'll excuse me using metaphors on this sensitive topic: if someone wants to be referred to as "His Majesty" but is not actually the king then while many nice people will indulge him[1], some who care a great deal about the "correct" usage of noble titles won't.
But what if you flip the coin? What if the person being asked really, really thinks that the other person is a pissface, and is a big stickler for calling people he perceives to be pissfaces as such? Surely we can all agree that allowing them to follow their ideas will lead to bad discourse.
And it doesn't even have to be such a crass example. What if a person is legally called Robert, but he's gone by Bob his whole life - is the stickler right if they insist on calling them by their legally given name?
> Surely we can all agree that allowing them to follow their ideas will lead to bad discourse.
I actually think that if someone genuinely thinks someone else is a pissface then it's their right to call the other person that. I also think that excluding such a person from your conversations may be sensible. This applies to pronouns too.
Sure, is somebody arguing for something differently? It's strange to me how often people repeat "oh, but they are allowed to", when that's irrelevant to the topic at hand.
It's the difference between "what you're doing is wrong" and "what you're doing isn't compatible with our vibe, so you're not welcome".
I think the main difference between my faction and the main trans activist faction is we don't think honest misgendering (done due to an earnestly held belief about the sex of the other person and some moral convictions against white lies) should be a firing offense.
> I think the main difference between my faction and the main trans activist faction is we don't think honest misgendering (done due to an earnestly held belief about the sex of the other person and some moral convictions against white lies) should be a firing offense.
This seems counter to what you wrote earlier:
> I also think that excluding such a person from your conversations may be sensible.
Let's say your a customer of a company, and an employee "honestly misgenders you", even after you've repeatedly asked them not to. What can the company and you do to exclude them from your conversations, without firing them?
Should a company also accept a racist employee calling customers the n-word? Is it acceptable to fire somebody for that?
This is where user options should allow for that. If being called a different gender is so bad just block the people that do that. The problem is you might need that word in general discourse as well. So you can't just create a rule that blocks everyone who says "He" just like you can block the n-word. The problem is with the person assuming every single person in the general public should treat them as a friend and getting offended when a label that is applied to half of the population (neither as a positive or negative, merely as a descriptor) is somehow that bad of a thing.
> This is where user options should allow for that. If being called a different gender is so bad just block the people that do that.
In your comment, you said that we can just block the n-word. Why? Should it not also be up to the users offended by it to block those using it? I don't think people using the word should be on the platform, but I don't know where you'd draw the line.
> So you can't just create a rule that blocks everyone who says "He" just like you can block the n-word.
I don't think I've ever seen a single person suggest this.
> The problem is with the person assuming every single person in the general public should treat them as a friend and getting offended when a label that is applied to half of the population (neither as a positive or negative, merely as a descriptor) is somehow that bad of a thing.
I'm not sure what situation you're presenting here. Trans persons usually don't assume that the general public should treat them as a friend. What they assume is that "if I tell somebody I would like to be referred to with female pronouns they should do so" is a very normal thing to ask of somebody. I'm male, but if somebody erroneously called me female, I'd correct them. If they kept calling me female, I'd try to get them removed from the social situation I'm in. Why is this any different for a trans person?
> (neither as a positive or negative, merely as a descriptor)
This is a pretty bad line of argumentation. For racist people, the n-word is also just a descriptor, and they don't understand why they're not allowed to use it.
In the end, referring to somebody in a way they don't like is always a demonstration of power. By doing so, you're saying "it is okay for me to make you feel less welcome, because I won't suffer negative consequences from doing so". The only harm in making people feel accepted is an imaginary one.
> In the end, referring to somebody in a way they don't like is always a demonstration of power. By doing so, you're saying "it is okay for me to make you feel less welcome, because I won't suffer negative consequences from doing so".
Well said. I know someone who is a vegetarian and their old boss always referred to them as vegan. Seems minor because neither word is offensive, but after correcting the boss multiple times, she realized it was just him asserting his power. A prick thing to do.
What does it matter to me if their belief is honest or not? I will still feel unwelcome, maybe even more so. Does it matter to a black person if the other guy really thinks "he is an n-word"?
>It's not about what I personally believe or think is unfair. It's about what Republicans (broadly speaking) believe.
No, it is about what Twitter believes. That was what I was referring to with point 2 in my original comment. Not every customer complaint is valid. It is ok to hear a complaint and dismiss it without further investigation. Twitter doesn't have some obligation to get all of society to think its rules are fair.
It is fine for a company to tell some potential customers to "fuck off" as long as that company isn't discriminating against a protected class. Twitter isn't discriminating against a protected class here.
If Twitter thinks misgendering people is wrong, it is impossible to come to an agreement with a group that think properly gendering people is wrong without Twitter compromising its own morals. Twitter is allowed to stick to its own morals and tell the people who disagree to "fuck off".
> Twitter's rules and enforcement actions behaved as if the leftist view of transgender people is the only valid and permissible view.
Twitter has changed in that regard since Musk took over. You can pretty much say what you like on trans issues now, as long as it doesn't break other rules. Loads of gender critical feminists have had their accounts restored in the past few months - usually having been suspended for 'misgendering' or some such nonsense.
> There is massive resentment from people on the right who think the Twitter rules unfairly elevated some political opinions as good/correct/acceptable while treating others as unacceptable.
Would it surprise you to find out that this resentment is in fact, conveniently manufactured, politically useful outrage? Because it's simply not true on its face, and the only thing we need to know to understand this is to see that it took Trump launching a coup to be banned on the platform. He violated the TOS every day, and he was allowed to spread his message to his millions of followers by Twitter. You want to talk about unfairly elevating political opinions? Trump used the platform to violate citizens' first amendment rights, and we had to take him to court to get those rights back. Twitter didn't do shit to protect us from him.
But it's not just Trump. It's right wing political opinions writ large. Far and away from sinking right wing conservative voices, Twitter research found they actually amplify right wing voices in every one of their top 6 countries except Germany [1]. Yes, that includes the US.
Is your mind blown? Have you heard of this once? I bet all you've heard from Musk and right wing politicians is that Twitter is going hard on conservatives and deplatforming them. Blocking their messages. Being unfair to conservatives and right wing opinions.
Yet what has actually happened? Twitter was actually deferential to conservative voices! It boosted conservatives and right wing voices at the expense of liberals. How did this happen? This is conservative messaging 101: complain about bias loudly enough and the other side will go so far out of their way to seem unbiased, they will be biased in the other direction. Conservatives managed to complain so loud about Twitter being biased against them that you not only believe it, but reality is actually completely the opposite.
Folks like Jordan Peterson would like you to think so, yes.
Of course, the man is essentially a walking “old man yells at clouds” meme at this point, so I’m not sure you should take anything he says with any merit.
Sometimes. Consciously using language that you know will offend another person and being uncaring of their feelings is shitty behaviour (unless you have a very good reason), using language that comes naturally and you think accurately describes the situation is neutral behaviour, making the effort to use validating language that people prefer is good behaviour.
There's also the communication accuracy issue. Words can carry a host of connotations and implications (e.g. pronouns carry information (in the information theory sense of shifting the probability distribution) regarding how deep someone's voice sounds) and using words that give the listener an inaccurate impression is a cost, however minor.
How much information is transmitted (and thus how big the "lie" is) depends on how different the male and female probability distributions (along many axes) are and how different they are to the transman and transwoman distributions. Which is obviously a very complex thing to figure out (are men and women identical other than outer appearance in X society? Obviously not, but how different they are is very unknown, for any X).
I don't know why people need to make this topic so complicated.
Imagine you meet someone named William. Maybe you become familiar with them and call them "Will" and they correct you and say they only want to go by "William". From that point forward, calling them "Will", "Willy", "Bill", "Billy", "Liam", or anything else besides "William" is rude. It doesn't matter that society at large thinks those names are all acceptable alternatives. It doesn't matter what is on this person's birth certificate. It doesn't matter the reason behind their request. An individual person told you their preferences and you ignored them. That is rude.
Pronouns are the same. You call people by what they want to be called. Anything else is rude to that individual.
The name analogy doesn't really work because in most societies names don't have very strong connotations. In ones where they DO have strong connotations (e.g. tribe/clan last names where different cland have different cultures) the analogy works, but most of us don't live in such societies.
If engagement on the tweets of that user goes down after a change has been implemented, you can roll back the change to prevent that user from being negatively impacted.
What if engagements around that user naturally declined, perhaps due to that user going off the deep end. Wouldn’t this just serve to bias the algorithm toward propping up the exposure of that user? Do they even care about the control so long as that user’s engagement is up and to the right?
Come on think this through. It’s trivial to tell the difference between a gradual and natural decline and a drastic decline immediately after rolling out a change. Especially when the change is rolled out region by region and only exists in regions running the update. You have to be able to measure the effect of changes and the most popular accounts are the obvious low hanging fruit for doing that.
I expect they're tracking the red team/blue team metrics because of the political shitstorm that's been the GOP's assertions they're being silenced by The Algorithm.
The fallacy of false equivalence systematized in code.
Now one side can spew as much disinfo and incitement to violence as it likes, and any algorithm change that prevents this shit from getting amplified will be rejected as bias.
This shouldn't really be a surprise to anyone. It was reported years ago that Twitter was unable to cut down on hate speech because the automated systems they developed triggered too many [debatably false] positives on Republican politicians and that was bad for the company's reputation. If Twitter wanted to prevent future code changes from impacting that approach, there needed to be something like this in the code or tests.
I don't know what specific documents you think did that, but "comprehensively" is absolutely an awful way to describe the Twitter Files. They were anything but "comprehensive". In actuality, they were an excellent example of how easy it is to lie using partial truths. For example, highlighting all the times Twitter took moderation recommendations from a Democratic campaign looks a lot worse if you hide any time they took moderation from Republican campaigns. A simple look at the specific journalists that were given access to Twitter documents and the strings attached to that access reveals that the Twitter Files were not about transparency. They were an ideological play and nothing more. If Musk wanted true transparency, he would have given wider access to more documents or just released them all like Jack Dorsey requested.
Could you please point out the list of moderation recommendations from Republican campaigns that were actually accepted and carried out? The only ones that were revealed so far were of the following types:
1. requests to un-ban or un-suspend right-wing personas
2. removal of explicit death threats.
3. Anything against the vaccine - even actual, scientific data.
From the twitter files, the Twitter team that was wholly left-leaning spent significant time and debate looking for the mildest of excuses to ban right-wing politicians, building black lists for them and even gloating happily as they managed to kick them off. Not much doubt about that - actually no one has even denied that. There were several hearings also in the house judiciary where they even confirmed the same.
(Interestingly, there was pressure applied on Matt Taibbi to either "shut up" or relinquish all his sources to law enforcement. Also, as a form of indirect pressure to rattle him, US tax agents visited his house the VERY DAY he would testify before US Congress stating that his tax returns had been rejected due to identity theft concerns - despite him having the electronic receipt which showed it being accepted.)
>Could you please point out the list of moderation recommendations from Republican campaigns that were actually accepted and carried out?
That is exactly my point. I can't because none were released. That is not a reason to assume they don't exist. You are assuming full transparency in a situation with only partial transparency.
WHEN THE WHITE House called up Twitter in the early morning hours of September 9, 2019, officials had what they believed was a serious issue to report: Famous model Chrissy Teigen had just called President Donald Trump “a pussy ass bitch” on Twitter — and the White House wanted the tweet to come down.
That exchange — revealed during Wednesday’s House Oversight Committee hearing on Twitter by Rep. Gerry Connolly — and others like it are nowhere to be found in Elon Musk’s “Twitter Files” releases
> there is also no reason to beleive they do exist.
Proof of reason to believe they do exist.
Furthermore, the office of the president, the FBI, the Government making a request to limit the free speech of a citizen is entirely different from Joe Blow in the street making a request.
If anything it gives credence to the theory that while both dem and repub lawmakers request Twitter to remove embarrassing tweets (obvious), Twitter pre-Elon only answered the dem's requests.
Twitter admits bias in algorithm for rightwing politicians and news outlets[1]
The research found that in six out of seven countries, apart from Germany, tweets from rightwing politicians received more amplification from the algorithm than those from the left; right-leaning news organisations were more amplified than those on the left; and generally politicians’ tweets were more amplified by an algorithmic timeline than by the chronological timeline.
According to a 27-page research document [2], Twitter found a “statistically significant difference favouring the political right wing” in all the countries except Germany.
Twitter admits it was boosting right wing accounts over left wing accounts. It treated right wing twitter users who violated its own terms of service better than regular users because of their position as conservative leaders. Trump himself was boosted and promoted while he violated Twitter TOS. It took him using Twitter to wage a coup against the US government for them to ban him. So yeah, there's more than a little reason to believe that there's evidence at Twitter of them boosting right wing accounts.
And PS: Notice that this research by Twitter is never mentioned by Musk or in the Twitter files. They're trying very hard to memory hole this report, and it seems like it's worked on you.
But that was published while the pro-Democrat establishment goons were in charge of Twitter, so of course they would say that because they wanted you to beleive they weren't favouring the left. It's hardly a reiable source. Same applies to Musk-era of course.
In the end, we accept the things we agree with, and discredit the ones we prefer not to believe. Human nature.
As I said, we don't need to know anything internal about Twitter to know for a fact that they bent over backwards to elevate extremist right wing voices. Because they elevated the most important extremist voice for 4 years as he violated the terms of use for their platform. In effect, we all had one TOS, and there was a different, more permissive TOS for the leader of the US MAGA-right wing movement, the most extreme version of right wing politics in America. That's blatant, confirmed, irrefutable, systematic bias and special treatment for conservatives on Twitter, pre Musk.
In case you want to discount the Twitter report as a false flag, here's a report from NYU that independently confirmed Twitter is biased toward conservatives. It also found the same for Facebook: https://bhr.stern.nyu.edu/bias-report-release-page
You really can't just blanket claim that all research from a university is politically biased, even if you feel American universities in general are biased. Come on now, we're here for discourse that's a little more nuanced than that I hope.
Since all you're bringing to the discussion at this point is a comic, I think that's all there is to say.
Yes, discussion over, you learned nothing, you'll carry on defending your fixed beliefs, and you got in a cheap comment about a serious comic. Well done.
I don't see an unbiased way to tell which "side" releases more disinformation and incitement to violence. Even deciding what counts as disinformation is hard (e.g. does it have to be literally false or just cause false beliefs in the reader?).
One way to tell would be to look at which side is incited to violence more often.
It turns out, according to the FBI (which is a conservative organization historically and exclusively run by conservatives), right wing extremism and violence is in fact the biggest domestic terror threat in the US, and it's currently growing [1]. FBI Director Wray gave this testimony after a right wing domestic terror attack was carried out that aimed to topple the US government. Not much has changed since then [2]. Since the former President's indictment the other day, the right-wing violent rhetoric has also ratcheted up a notch, so we can expect right-wing violence to follow.
Notably, we can confidently say this doesn't happen on the left, as when Hillary lost they did not launch an assault against the Capitol as the right did. Instead, they knit pink hats and had a march.
(PS before anyone whattabouts the George Floyd protests, the FBI doesn't see them the same way [3])
I clicked downvote before reading the whole comment. At first I thought you were talking about the red team when you started with "mass riots and violence..." Then I read the rest of your comment and still felt just as good about my downvote. This isn't a constructive comment no matter what "side" you're on.
“Without debate, without criticism, no Administration and no country can succeed--and no republic can survive. That is why the Athenian lawmaker Solon decreed it a crime for any citizen to shrink from controversy. And that is why our press was protected by the First Amendment-- the only business in America specifically protected by the Constitution- -not primarily to amuse and entertain, not to emphasize the trivial and the sentimental, not to simply "give the public what it wants"--but to inform, to arouse, to reflect, to state our dangers and our opportunities, to indicate our crises and our choices, to lead, mold, educate and sometimes even anger public opinion.” JFK
Everybody remembers January 6 except those who want to pretend it didn't happen.
How many remember the floods of Twitter incitement to hit the gas in their F-150 trucks to run over protesters, and then how many people actually perpetrated vehicle attacks?
They don't have to, because he effectively can't be muted. People tweet quote him with an image, and it's not blocked even though I have him blocked as an account. This behavior is pervasive enough that you can still see his tweets all the time.
I imagine it's the largest metric in a mission control style room
it starts dropping, klaxons start blaring, the room drops to red only lighting, engineers on the floor start pulling out their hair knowing the shitstorm that's coming
Per original source, The code that was released today doesn't show the parts that actually alter the scores of Elon and other users. The part of the code referenced below just tracks Elon stats (from what I know). Employees removed most PII before the code was released.
Update: Elon was asked about these in a Twitter Space, he says it's not appropriate and will be removed from the codebase.
Additionally, from another Twitter engineer, the Democrat/Republican flags are apparently 10 years old and not important and do not have high feature importance.
Elons reply “I only learned about it now!”. What a crock of shit. We literally went through reporting a few months back where he was clearly instructing the team to make sure his tweets always come up, for everyone.
If I remember correctly, he noticed that his account with 100,000,000 plus followers was only getting 40,000 or so views. This lead to a Twitter engineer saying maybe his content wasn't interesting enough to get views and getting fired. And later on Musk said that they found a bug in the fanout system because his account was so big it basically broke things and they fixed it.
I remember everyone agreeing with the Twitter engineer who said maybe his content wasn't interesting enough, while to me that seemed odd. If people follow an account and they tweet and they're online they should have a high chance of seeing that tweet. That's the entire point of following someone. If someone I've followed tweets something I would like to see it.
The fanout bug was in weighting how blocks affect visibility and fanout.
Apparently it was an absolute and not a relative thing, so the huge amount of blocks musk has affected their visibility.
You don't, at least for me, I have 5700 followers and most posts (with art) only see 200-500 impressions, whereas a year ago I was getting 3000 on average. The Following tab is clearly (at least for me) not in timeline order, so most of my followers never even know I posted anything. Since I post art most of my followers are artists, so you would think they would see them. Impressions (which are Twitter putting tweets in a timeline so that people scroll by them) are not under the control of the poster. Likes per impression for me are up, so interaction is higher which is supposed to increase how often tweets are shown, but only timely retweets seem to help any.
I am trying to figure out how the algorithm decides on what to show in the Following tab, but the code is way too big to analyze without being able to run it and look at logging/metrics/stats.
Yeah but presumably he didn't think the devs would tag a specific metric for "elon", which would later be open-sourced. It's one of those things that makes more sense in retrospect...
As in: "My tweets are very important, if they don't show up on top, it means the algorithm can't recognize what is important, it needs to be fixed". And the team, who probably didn't see in which way Elon Musk's tweets could be that important besides the fact that he wrote them, they just decided to give Elon Musk's tweets a boost.
The most hilarious things to me was when I was reading some (unrelated to this drama tweet) posted in an article and the top recommendation was Musk's tweet that he doesn't artificially promote his tweets.
From what I understand of upthread, it doesn't "give elon musk's tweets a boost", rather any change which downgrades musk's visibility is considered a regression.
Obviously the end result is similarly that musk's visibility can never decrease, but it's a more technical (and to the letter) compliance with the specifications.
> We literally went through reporting a few months back where he was clearly instructing the team to make sure his tweets always come up, for everyone.
Feel free to give me a source for that, but I'm pretty sure that's not true.
It was widely reported in the days following the Super Bowl that he personally ordered his Tweets be boosted after a Tweet by Biden got more engagement than his similar Tweet [0].
As far as I can tell, he did not instruct Twitter employees to make his Tweets appear to all users, but he did want them to make his posts appear in timelines significantly more often. This lead to reports of users suddenly getting their timelines flooded with lots of his Tweets.
It seems like there was a rush within Twitter to raise Musk’s engagement numbers by altering the recommendation algorithm to specifically boost posts from his accounts. The special boost factor was later reduced, but allegedly still exists.
Wrong. Elon noticed his tweets had low engagement relative to his follower count compared to other accounts.
After some digging in the code base it turned out that there was a de-boosting factor based on the absolute numbers of blocks your account had, which affected popular and controversial accounts (like Elon’s) unfairly.
This investigation was initiated by Musk but it resulted in a great improvement in the algorithm with no special treatment for his account.
Well that's what the claim was. I never bought it, to me it sounded very much like the "Dear Leader Kim observed during the factory tour that one of the machines was miscalibrated and that productivity could be increased tenfold". Remember that he'd just fired an engineer on the spot for suggesting there wasn't a technical reason behind his lower engagement, and that the unspoken reason was he's frankly just a bit annoying and his epic meming CEO shtick was wearing thin. So I think when he raises the issue again, they just implement `author_is_elon` and they get to tell him that actually he is good at posting after all and there was just a weird technical issue they have now fixed.
Elon's happy, the engineers he turned to kept jobs and any time I find myself in the "For You" tab I see Elon's memes and his @catturd2 RTs.
> By Monday afternoon, “the problem” had been “fixed.” Twitter deployed code to automatically “greenlight” all of Musk’s tweets, meaning his posts will bypass Twitter’s filters designed to show people the best content possible. The algorithm now artificially boosted Musk’s tweets by a factor of 1,000 – a constant score that ensured his tweets rank higher than anyone else’s in the feed.
> Internally, this is called a “power user multiplier,” although it only applies to Elon Musk, we’re told.
It seems indisputable that Musk’s account gets special treatment, even if some of the changes may also boost other controversial users.
You did not reference any sources, while the article claims sources inside Twitter gave them this information.
Anonymous Twitter sources may have _low_ credibility, but Elon musk has _exactly zero_ credibility at this point. He has a significant and proven track record of making false and misleading statements, even ignoring the many many more difficult to prove allegations.
In my personal experience, software engineers, when you ask them about technical matters like "does the code do X" or "is there a condition for Y" are generally pretty honest. Programming is a task that requires being able to have structured and literal reasoning, and asking engineers about purely technical details usually triggers that response, even if you can imagine a social motive for lying.
Disagree, Elon has high credibility IMO, certainly higher than an anonymous source with incentives to slander. Not saying he is perfect but given the scrutiny he is under and how much he is “in the arena” pushing the envelope.
Elon is furthermore a software engineer (he wrote ALOT of code at zip2 & paypal) so your argument falls flat in that regard.
Credibility from what exactly? Having last year's most expensive mid-life crisis? He once had some credibility, no one will deny that. Respect is earned and lost.
I think him buying twitter was great. He is lightyears better than previous owners & management. He is also running two other companies doing extraordinary things in the world.
Of all the people you can hate on in the world, hating on Elon is to me very odd. Tall poppy syndrome I guess...
You're making it sound like this commenter has some nasty personal vendetta on Elon, when it's just that Elon's a goofy dude who keeps embarrassing himself in a very public way.
This isn't Tall Poppy syndrome. This is Sideshow Bob repeatedly walking into rakes and getting smacked in the face.
He makes way more mistakes than I would like as well. But he also has made a lot of fantastic decisions that IMO outweigh by magnitudes the annoying irreverence he displays.
By the way, I’m not excusing his behavior by saying this, but I’m pretty sure the irreverence is a complete response to the struggles and pain from non-stop attacks from fighting giants in the arena for years. Putting yourself out there, living in the arena, especially to the extent he has, is super hard. Most of us don’t have the courage. If you do, you’re welcome to enter it and try to be a better role model. The world needs it.
He tried to fire a disabled employee via a public tweet without speaking to them privately first, and then had to retract it. This is McDonalds McManager behavior, not CEO behavior.
Musk did not write a lot of code at PayPal, that's factually incorrect. And he was ousted and replaced at PayPal because he was trying to make technical decisions that would've killed the company.
Note that some of the "false" statements later turn out to be true. E.g. holding child a last heartbeat happens (ex-wife held him for death rattle, elon for last heartbeat)
While it was thrilling for a moment to imagine myself as a disgruntled former Twitter staff member, that is not the case and I have not been employed there.
My statement above was to read ’we’ as in as the tech community not ‘we’ Twitter staff; others have quoted some of the sources on this thread, but it was definitively reported by a few different outlets at the time.
I think the decade old comment related to a different part of the code regarding the number of followers you have in relation to the number of accounts you follow. (Everybody on the call wants to remove this: I wonder why they haven't yet.)
Chesterton's Fence. In a sufficiently large system, you should be hesitant to remove things unless you're sure you know why it was added, and all the things that have come to depend on it since.
I've definitely been hesitant to remove things I was pretty confident weren't used anymore, just because I didn't want to deal with the repercussions if I was wrong.
I didn’t know there was a name for this. Thanks for sharing.
I’ve definitely been bitten by this. You always have to weigh the chance you break something against the upside. If you’re actually fixing a bug, fine. But just refactoring to make something cleaner? Or deleting because it seems like it’s not doing anything, even after doing some research? Think again.
The author_is_elon flag doesn't surprise me, but the two political designators are somewhat shocking. I'd sure like to know what changes based on what Twitter knows about your political affiliation.
But on the topic of Democrats vs. Republican vs. independent; a big factor may be that "Democrat" and "Republican" are much more cohesive groups and therefore much easier to define. No one can honestly define "independent" except in a kind of "none of the above" sense, since they can range anywhere from extreme right, to the center, to the extreme left.
You might want to ask that of, say, astronomers, paleontologists, or even simply journalists.
Observing and measuring leads to understanding. As others here have noted, sometimes you want to measure to ensure that you're not inadvertently affecting an outcome or phenomenon.
> You might want to ask that of, say, astronomers, paleontologists, or even simply journalists.
You compare scientists with businesses. One of them's job and passion is to collect knowledge for the sake of knowledge. For the other one it would be cost without gain and eliminated if they didn't do anything with it!
No idea what journalists are doing in that list, what do they measure? If they want something measured they'd ask or look at other's services. Unless you mean the business that the journalists work for, but hen it's just that, a business.
I gave examples based on well-understood instances for which the reader is presumed to have the capacity to draw inferences to business-related concepts.
Businesses also rely on astronomy and geology, in instances. The former is used for navigation (though far less than in the past, I'll grant), and there are certain extractive sectors with interests in geology. Risk-management as well (insurance and catastrophic risk, whether from landslides, earthquakes, volcanos, tsunami, or other phenomena).
You'll also find businesses keeping tabs on weather, climate, competition (competitive intelligence), demographic data, politics, legal cases, laws, social and cultural movements, etc., etc., etc., which in many cases they have comparatively little capacity to change directly.
You're also jumping late into a thread which has already given numerous other rationales for why such activities might be undertaken.
Sometimes approaching a discussion from the PoV of seeing what you can learn from it rather than automatically adopting a presumptive stance of opposition or disagreement affords benefits. I recommend it strongly.
Depends what the metrics are used for. It doesn't make sense to apply artificial boosts to metrics that are only used for internal accounting. Well, maybe if you have an egotist CEO, but that wouldn't explain the rest of the boosts. We have to assume this code has some sort of effect somewhere.
This is what it means to let another country own your social media. Their ideas and memes unconsciously get preferential treatment. This is maybe not a good thing, but it is what it is.
It’s omission. They built for Americans because they were Americans. No one building this said “let’s ignore Canadian politics” or any other country, they just didn’t think about them at all, because like most Americans they don’t really care about the insidious Québécois plots to annex Prince Edward Island or whatever actual issues are happening in Canada.
The freedom convoy and it's repression with suspension of constitutional rights (because that's a thing over there?) and bank accounts arbitrarily frozen did get a little bit of attention in the American medias.
> the insidious Québécois plots to annex Prince Edward Island
I know it's a joke, but the people of Prince Edward Island would most likely welcome it. Economically that would be a huge boost. Not sure Québec would enjoy it however.
> The freedom convoy and it's repression with suspension of constitutional rights
This was needed because the convoy of morons was seriously damaging people's quality of life and the local police did not do their job. The police in other towns DID do their job and blocked the trucks. You should be mad at the cops who just sat by and watched the convoy of morons roll in with their thumbs up their butt.
Indeed their acts of terrorism were not reported by international media.
For example, during the occupation of residential areas of the capital they created deafening noise around the clock that was measured indoors (windows closed) at levels causing permanent hearing damage after minutes of exposure, with police doing nothing to stop them.
People can always use their countries' social media (and I assume many do!).
But it's indeed an interesting fact that people seem to specifically seek American tech and social media. And honestly, there's no shortages of foreign nationals commenting on American politics (and it's a good thing, it's their right thanks to the first amendment!).
The vast vast vast vast majority of people use “American” social network because they are the social networks that exist. The US is undeniably the main exporter of SaaS products. See: the incumbents freaking out that China is getting a turn in the front seat. I’ve never once heard of anyone seeking ‘American tech’, except for some punchline in an anti-Soviet joke or movie.
Foreign nationals can comment about American politics because the US doesn’t have jurisdiction over their speech, notwithstanding the back and forth over whether social media companies are liable for disseminating such content in the first place.
A material part of why the USA is seen as The Country in Western culture, and a noted big player in other cultures, is because of the power it projects via the media. That includes both Hollywood exports, and social media.
American social media companies have the benefit of a large homogenous deeply interconnected culture with a vast ad buying community. Simply put, a U.S. social media platform can get more revenue with less effort than a European counterpart (because they have to go country by country, language community by language community), and will generally outcompete those EU counterparts, with social network reinforcement effects doing the rest. This holds up for most ad-supported online services, which together with a much stronger private investment sector is the reason American platforms absolutely dominate Europe. The EU has a stronger government subsidy system for software but that system does not reward or expect market success, which is why it delivers very little value.
> I’ve never once heard of anyone seeking ‘American tech’, except for some punchline in an anti-Soviet joke or movie.
Really? Why is everyone using MacBooks and iPhones? Actually most Soviet computers were copies of western designs. [0]
> is because of the power it projects via the media. That includes both Hollywood exports, and social media.
The thing is, is takes to to project, someone has to export a product and someone else has to import it. Since people seek it and want to consume it it's easy to export.
> I’ve never once heard of anyone seeking ‘American tech’
One reason people have a distorted idea of what physical products the US produces, so I've read, is that most of them are products for industrial use, in factories and so on.
You look at consumer products that are made in..., because that's where the finishing touches are put on. That's not representative of the global economy though.
Before I deliberately locked myself out of it (well before Musk), I asked for my data.
They classify me as:
* speaks Indonesian
Interested in:
* Beer
* Cricket
* DJs
* Dance
* Enterprise software
* Horror
* NFL football
* South America
And aged either between 13-54 or (and?) over 65
Other than the age (I'm neither under 13 nor between 55-64), everything I've listed is incorrect.
On that basis, they'd probably call me a Republican.
Carries a different meaning when you're British, that name does.
What a strange age classification — no idea about what your age actually is, just that it definitely isn't 55-64. I assume it would never conclude you are under 13 for legal reasons.
But ever since Musk took over the amount of US political content has significantly increased in particular from the right despite me not living in the US.
It's hard to tell whether previously political content was weighted less and Musk has removed those controls or whether they are now weighted higher.
Some of the power users Musk reportedly had boosted are specifically right wing political posters, like catturd2. But then he's also boosted some high profile left leaning politicians, so it's not exclusive. It does mean you're more likely to see right wing American political content either way, which has to be annoying for people outside of the US.
I believe their point is that, because the US is generally regarded as 'to the right of' many countries in Europe, for example, a slight rightwing bias looks to us (I'm in the UK) like an extreme rightwing bias. Heck, even a completely neutral stance, as far as the US is concerned, will look like it veers to the right for us. Note that I'm not saying anything about how I believe the twitter algorithm should work, this is just my interpretation of what that commenter meant.
Europe is only a tiny fraction of the world, though.
Compared to India, Japan, Singapore and (depending on the metric) China, the US looks clearly left of center. Compared to most of the Middle East, the US looks extremely left-wing.
That extreme polarisation didn't start with the last few presidents; it's been the explicit strategy of the Republican Party to demonize the Democrats in every possible way ever since Newt Gingrich became a prominent figure in the 1980s.
It was frustrating to see Obama keep reaching across the aisle and basically getting spit on. Democrats have their faults, but Republicans have made it clear that it's impossible to reach any kind of compromise with them.
If these ‘distraction-causing arguments’ are affecting real people (read: foreigners, queer people, black people), then they’re worth paying attention to regardless.
Is your assertion that this is all some big conspiracy how you justify not caring?
No, it's that people have bought so heavily into just attacking 'the other' instead of trying to get along and being critical of policy and authority instead. The distraction is the direction to argue with each other rather than against policy/corruption/authority actually causing the suffering
some of those opinions are that others of certain sexual orientations are indeed subhuman and/or should be treated as much, and due to Popper's paradox of tolerance, we don't need to, and shouldn't, tolerate such intolerance
That's one interpretation, which I see used as justification to attack and be generally horrible to anyone who disagrees. I see a lot of commenters making comments that Republicans are some kind of lower intelligence being too (same the other way too). Are these people allowed to be offended?
Unfortunately the plight of LGB (and Womens' rights to an extent) has been utterly brushed aside by the continual addition of each letter to the cause and the arguments too hot and aggressive to separate out the underlying causes for fair debate on their own.
> Republicans are some kind of lower intelligence being too
You HAVE to be pretty stupid to fall for the really blatant propaganda the GOP uses. Remember that scary migrant convoy they invented just before an election to use to scare people into voting GOP and then it just vanished right after? Have you heard the incredibly stupid things Tucker Carlson, MGT, Matt Gaetz, and Boebert say? People like Carlson are way too smart to actually believe the crap they spew, they knowingly spend their lives telling lies to gullible and/or stupid people to manipulate them.
> Of course there is no left wing propoganda to fall for though, right?
I'm honestly not aware of similar propaganda on the left. By similar I meant an entire industry designed to lie and gaslight people into voting for people who will cause them direct economic harm. The GOP has made lying an entire industry and an art form. Now they are demonizing trans people in a way extremely similar to the way Nazis demonized Jews.
> Do you wonder how you came to form such blatant negative opinions of your political opposites
I know exactly why I have formed such a negative opinion of the GOP, because I've watched them lie blatantly and egregiously for the last 20 years and I'm really sick of it.
whataboutism doesn't really address any of the points in the post you're responding to, and certainly didn't answer the questions in that post which were posed to you
Because I don't disagree with those points. But the idea that there is nothing 'noticeable' from the left shows they're a) maybe better at it and/or b) saying things that are believed at face value because they align to beliefs.
For the other commenter to be so entirely blind to their side's propoganda should at least cause pause for reflection.
your supposition regarding a) and b) are pretty unconvincing, they're on the level of conspiracy theorists who say that a lack of evidence is just stronger evidence of the conspiracy
frankly your claims of equivalency are pretty unconvincing, too, but that's neither here nor there, since the topic is one specific side, not "both sides" or how you personally think they're equivalent
anyways, let's get back on topic to where we were before the whataboutism reared its ugly head (and we can both just pretend like you haven't been attacking the commenter themselves): https://news.ycombinator.com/item?id=35397668
you say "attack and be generally horrible to anyone who disagrees", but recall that the thing said "attackers" are disagreeing with is equal or greater animosity towards people on the basis of their sexuality versus their opinions
in other words, it's a reaction to intolerance, specifically a decision not to tolerate intolerance - as we covered before, we don't need to, since doing so will end tolerance entirely (see Popper's paradox of tolerance for this)
I think Popper's paradox does nothing really but talk around an issue by changing the argument. Intolerance is not black and white, it is contextual. You can be intolerant of those who break the law, because they break the laws that hold society together (at a basic level). Paradoxes often aren't so problematic if you take them out of their own frame of reference (e.g. Zeno's paradox & Achilles).
I think the context we are referring to here is really free speech. Should we tolerate speech (opinions) that we despise or disagree with? Absolutely. And here we have the attackers responding to feeling insulted (framed as 'intolerance') by not being accepted/agreed with. Then they respond in kind and no progress is made.
You show me those people responding to equal/higher hostility and I guarantee I can show just as many examples of the opposite. And what I mean there is not "you're side is worse," I mean there are aggressive and horrible actors on both sides, who should not be the frame of reference for debate. But too often they are, and too often examples of hostility as used as an excuse for escalation or to avoid real debate.
It's nice that you personally think those things, but none of those things you personally think override or disprove Popper's paradox of tolerance, and thus none of those things you personally think will convince people to tolerate the intolerance that is, just as a single example, right-wing anti-trans intolerance
indeed, responding to, say, right-wing intolerance of trans people, by not tolerating it, is totally okay, despite your FUD, for reasons described by Popper
don't like your own intolerance of e.g. trans people, and actual attacks on them, to be responded to? maybe keep your intolerance and actual attacks to yourself, problem solved
This is an interesting point... they're making sure their A/B experiments don't adversely affect Republicans or Democrats. But lots of European countries are skewed more to the left and would broadly look more similar to the "Democrats" group.
Most big companies I've worked at are international and are regulated in many regimes, although sure the one with the main corporate headquarters is more important.
Or anti-statists! Not everyone who engages with politics is a bootlicker for authoritarianism but it sure feels like there’s no space made for this perspective (obviously)
Isn't it intrinsically self-reinforcing, if you have a winner takes all system? It's almost always better to join an existing team than start a new one.
It isn't an attitudinal problem, it is the logical outcome of our political systems. In political science it is known as Duverger's law: single ballot, winner take all systems inevitably tend toward a two party equilibrium.
Changing this requires states to adopt alternative systems, which can sometimes mean amending state constitutions. It isn't easy or straightforward, and the general sense is that there are better things to spend that effort on.
Parties come and go, but there will be two major parties. In fact under such as system if you're in one of those two big parties and you see a third party rising you need to figure out whether it's you or the other guys getting replaced, 'cos it won't stay a three party system for long.
For example in the UK, the Tories ("Conservative and Unionist Party") and Labour are currently the biggest parties, but a hundred years ago this was a novel situation, Labour were seen as a third party, while the Liberal party (which was eventually absorbed into what is today "Liberal Democrats") had seen success over decades and were often in government prior to that point.
A lot of it depends on whether your governing coalitions are formed before or after the election.
The US parties are just coalitions of disparate interests joining together until they (maybe) represent enough people to have a majority and be able to enact their collective interests.
I mean we're on the sixth "party system" in the US too, if that's your standard (it's a new "party system" when there's a significant realignment of which interests find themselves in which parties, either by a reshuffle among existing parties, or by new parties rising and old ones falling). Have you heard much about the Federalists or Whigs lately?
The voting demographics change frequently as well. California has voted to elect more republican presidents than it has democrat ones. It’s voted democrat in the previous 8 elections. In the 10 elections prior to that, 9 times it voted for republican candidates. Texas has also voted for more democrat presidential candidates than republican ones. People who think the US democracy is a rigid and highly predictable system simply have a recency bias.
It's just different order of operations on coalition building. Other systems divide into a majority and opposition at some point. In the US it just happens earlier, but there is the same diversity of opinions within those groups.
You'd think a country that played a central role in a global, decades-long unstable regime of bipolar power that routinely pushed mankind to the brink of nuclear oblivion would know better than to have a bipolar electoral system
have you never heard of the many times we almost came to setting off full scale nuclear warfare because of the bipolar power war of the ussr v usa et.al.?
the only thing that saved us was cooler heads that prevailed on both sides.
Why would throwing more actors with similar capabilities into the mix make the situation any more stable though? That seems like basically the old European Balance of Power, which broke out into open conflict more frequently.
This balance is the central theme of the seminal work in the field (https://www.jstor.org/stable/j.ctt5vm52s). It may or may not be correct but is one of the most influential texts on the subject.
I take the view that the European balance of power probably broke out into open conflict often because of hidden alliances that made it hard for states to correctly gauge the costs of engaging in such conflict.
In the field International Relations, there's lots of discussion around the stability (or instability) of a bipolar distribution of power. Your stance is closer to neorealists, mine is maybe closer to classical realists. Have fun reading https://en.wikipedia.org/wiki/Polarity_(international_relati... and related links
But from a game theory perspective, having only two powers turns everything into a zero-sum game. I argue that leads to less cooperation and increased divisiveness. Not agreeing with Republicans means you must be a Democrat in their eyes. This "us versus them" mentality is somewhat tribal and leaves little room for nuance.
If I had a magic want, the US would have a 4 party system. Like a cartesian plane with civil liberalism <-> conservatism on one axis and economic liberalism <-> conservatism on the other axis
> Classical realist theorists, such as Hans Morgenthau and E. H. Carr, hold that multipolar systems are more stable than bipolar systems, as great powers can gain power through alliances and petty wars that do not directly challenge other powers; in bipolar systems, classical realists argue, this is not possible.
I suppose they must account for this somehow but isn't that exactly what a series of proxy wars in far-flung places between the United States and the Soviet Union were?
It's been almost 20 years since I studied any of that, but those proxy wars in far-flung places did directly challenge other powers because of the Domino theory
I don't necessarily agree with Morgenthau and Carr as I think most IR Theory is bullshit made up by academia... particularly the stuff around how players "gain power"
So I make my own argument which mostly hinges on the idea that two powers really means "my power" vs. "everyone else" which is not a recipe for peace
That's all nuclear incidents ever, not just US/USSR. US/USSR count from that page is more like 10 incidents across 40 years. Some of them occuring within days of others, they're that fine-grained.
In some other countries the different interest groups sort themselves into two factions after being elected but I don't know that it is really that different in practice.
It's not different, or rather it doesn't produce meaningfully different outcomes. I'm not aware of any parliamentary system with a wonderful diversity of thought and a long record of positive accomplishments. You end up with ruling coalitions which are typically pretty awful.
Ironically, America has one of the most open political systems. You register as one party or the other and vote in primaries. This has lead to a huge variety of people replacing hated mainstream politicians. That's way more than you can say for many other countries.
>it doesn't produce meaningfully different outcomes
It absolutely produces different outcomes than a two party system. Smaller parties make demands as a condition for joining any coalition (or similar arrangement). For example, Canada's NDP only agreed to back the Liberal Government on condition of state funded dental care for children being implemented. Now it is. Millions are affected. Whether you agree with it or not - that is unquestionably a "meaningfully different outcome". Other examples abound if you care to look.
>I'm not aware of any parliamentary system with a wonderful diversity of thought and a long record of positive accomplishments.
That's an impossibly high bar. The standard we're talking about is whether it's better than a two party system.
> It absolutely produces different outcomes than a two party system. Smaller parties make demands as a condition for joining any coalition (or similar arrangement). For example, Canada's NDP only agreed to back the Liberal Government on condition of state funded dental care for children being implemented. Now it is. Millions are affected. Whether you agree with it or not - that is unquestionably a "meaningfully different outcome". Other examples abound if you care to look.
You think logrolling doesn't happen in the American system? Parties are made up of factions who will entertain each other's preferred priorities, which is the exact same thing you're describing except that the parties of the various factions are nominally the same. There's no real natural or obvious philosophical reason why your position on gun ownership should imply a position on environmental regulation, religion, infrastructure buildings, racial politics, abortion, tax policy, and more. We're so used to the groupings that they seem natural but if we go back and looking at older American party systems you'll see parties that don't 100% map onto the contemporary ones, with a blend of some elements we would think of as fitting and others we wouldn't.
It's unquestionably a different dynamic. If the party is in power, then the party is in power. Period. They have less incentive to listen to smaller factions. Do they? I sure hope so. But it's not the same.
Furthermore, these internal deals are more likely to be kept private since theres no benefit to airing everything to the press, the public and the opposition.
In a multi-party system almost all those deals are public. Thus, voters can decide if everyone lived up to their promise.
Anecdotally, a recurring theme in conversations I've had while living abroad is the desire to prune or consolidate some parties.
While both sides in the US have big tents, they are effective in whipping votes when things need to get done.
It also helps that detracting coalition partners can't torpedo their leadership. Historically, factions within a party, like Blue dog dems or Tea partiers, had to wait for an election to litigate their grievances.
Well, yes, this was where I was going. The coalition is just formed before the election takes place. I live in a city with "nonpartisan" elections and I feel like it mostly just makes it difficult to understand what candidates in local elections even stand for.
In some ways it might be worse. Here we had a party that promised to vote with one faction and partly thanks to it survived the election. A few months later they did a 180 and voted the other faction into government. Opinions aside, imagine voting for Sanders only for him to elect Trump or the other way around. With that said, two party version also comes with major flaws.
Majority? Probably. Vast majority? No way. If that were true we'd not keep switching parties between presidents. And Florida, a mostly independent voter state, wouldn't have had DeSantis win in a landslide when he just barely tied last time around.
So many questions. How are users tagged D or R? Is that a manual process or automated somehow? What is the effect of these tags? Can I find out if my Twitter account is in one of those buckets?
you could probably algorithmically determine it in most cases based on any number of indicators from phrases used, to communities interacted with, which hashtages are included, which cohort retweets and likes most etc... thats not even getting into simply tagging political figures with the party they officially affiliate themselves with
Likely they are tracking performance verified politician accounts based on registered party affiliation. Why republican should count equal in the evaluation metric to democrat when nunerically there are less republican voters, let alone proportions on Twitter, is another question.
And how are they choosing to balance them, per capita, or just both sides should get 50%? It seems pretty clear they are making editorial decisions here. Does that break their section 230 protections?
I suspect that these are used for metrics tracking rather than being fed back into the recommendation engine. But there's no real way to know for sure given the limited release. These predicates aren't actually used anywhere in the code that's been made available.
Half the people that got promoted on my timeline were perpetually candidates for elections I couldn't vote in, and they _self-identified_ as Republican or Democrat in their own bios, or via the registration of their candidacy...
This is why I exclusively used to use Twitter in the "people I follow only" mode, and simply shut my account down when they pushed harder on the algorithm.
Why specifically track political parties? Where is author_is_american? Or author_is_mayonnaise_enjoyer?
Maybe it was a choice made many years ago that they thought was appropriate, but we can't yet know it's not used for other purposes. We can at least be reasonably sure they've added the author_is_elon within the past year. I would have thought there would be many more descriptors, or non-controversial descriptors.
Or maybe Elon specifically added those before releasing this code to get people riled up.
I don't believe echo chambers are nefarious - there's no hidden agenda involved with them. That's just how recommendation algorithms work, and it's what most people want.
But if someone finds some code that suppresses recommendations from a specific political ideology across the board, that would be nefarious, IMO.
They can be if they're involuntary and inescapable, but neither is the case for Twitter. It's designed around letting you curate your own feed, but it also constantly throws random stuff in through retweets and quote tweets - which is what people hate the most about the platform.
I don't believe repeating someone's comment nearly verbatim is as clever as you want it to be, nor do I believe recommendation algorithms are equivalent to the kinds of societal discrimination you allude to, nor would any reasonable person.
Instead of trying very hard to be clever, please next time try just as hard to make a valid point. I know it can be difficult when you think you smell karma in the water but do try.
Did you not get the memo? "If you're not with us, you're against us".
It's probably more of a conservative/liberal identifier based on US political party ideals... And they likely would filter any metrics from this by the users country
Well someone just asked about it in the live spaces[1] Elon is hosting and he said that should not be there. An engineer said afterwards it is just for metrics but then Elon chimed in again and said "we should get rid of it, it should be gone."
There are millions of lines of code existent. There are thousands added. Your prior for it being Elon's addition should be something like 10,000/1,000,000 or roughly 1/100. The prior that it wasn't Elon's change is going to be something like 99/100.
When you add the additional information that Elon wants the code removed, but existing Twitter engineers think it appropriate to keep this actually increases the probability of it being added by the existing Twitter engineers and decreases the probability it was added due to Elon.
Obviously, these are rough numbers, but hopefully seeing any numbers at all helps you to get an intuition for the math.
Why is lines of code the appropriate input here? Here's a different computation that is at least as plausible:
There are hundreds of millions of users. Let's say 300M. Only a single one is special-cased in this code: the narcisistic CEO who reportedly went ballistic when his engagement metrics went down. The prior that it's a change done in response to his demands is 299999999/300000000.
(But of course it was added by existing Twitter engineers. The odds of Musk being able to actually commit code to their repository are zero. Even if he had the permissions, the man simply does not have the technical acumen to make even a trivial change.)
I think you are right, my estimate is much much too low.
I'll explain my mistake and why I made it.
I think I thought to use the estimate I did because Elon claimed he didn't know. The prior probability of not knowing something in a code base with millions of lines is very high, but contingent on his involvement in the change the prior that he is aware of it is much higher. So I started the estimate attempt with the probability that I thought better predicted the production of evidence claiming he didn't know.
Your point does raise my estimate substantially, but I think it probably raises it less than you would expect. I don't agree with your 1/300M prior, because I'm aware that hot users get special treatment. I've seen Elon's account thrown around in interview-style questions about hot users before and used as an example of a hot account that needs special treatment. This is something I've witnessed, but it wasn't contingent on Twitter being acquired and it happened prior to Twitter being acquired.
I also don't particularly assign high odds to wanting it, based on the evidence that he claimed to not want it implicitly by wanting it removed. I don't think it seems appropriate to get to near certain probability the he wanted it with the evidence being that he stated that he didn't want it. In my view there isn't a compelling reason for him to lie about this. He owns Twitter, so if he wanted them to have his account monitored that would be a reasonable thing well within his authority. If he wanted it, he doesn't need to pretend to not want it in order to appease someone.
It does seem to me that the odds that the change was added in response to someone thinking he wanted it is much higher than 1%.
> I don't agree with your 1/300M prior, because I'm aware that hot users get special treatment.
That's absolutely fair, and 1/300M was a reductio ad absurdum rather than a serious proposal. Not all users are equal, just like not all lines of code are equal :)
I have a few issues with the "hot user" theory, but they all boil down to the same point: no matter what the use case, you'd never want to do this with a single static user.
Does your infra require special-casing for accounts with more than 100M followers? That should be a flag in the account properties that gets flipped manually or automatically: if these users cause infra problems, you really don't want to be making code changes + full rollouts whenever a new user becomes hot.
Is this just a guard-rail metric, to make sure there's not some bug specifically affecting hot users that tanks their engagement? You'd want a much larger static set than a single account just to ensure there's a large enough number/variety of tweets to compute metrics from. A single user might take a break for a week, or might only be posting very specific kind of content for an extended period of time.
In any case, even if you chose to do this with a single user rather than a set of users, why would Musk be the obvious single choice? He wasn't the most followed Twitter account until two days ago. A year ago there must have been at least a couple of dozen accounts roughly as notable as Musk. The odds of him having been chosen as the special case still would not be very high.
> In my view there isn't a compelling reason for him to lie about this.
The reason to lie about this is that it makes him appear weak, needy, and a target of even more mockery. Given the purchase of Twitter seems to have been a vanity project, having this be exposed and leaving it in goes directly against his apparent goal.
> The reason to lie about this is that it makes him appear weak, needy, and a target of even more mockery. Given the purchase of Twitter seems to have been a vanity project, having this be exposed and leaving it in goes directly against his apparent goal.
I think it only makes sense to think like you are if you've adopted equilibrium assumptions; if you haven't then I find this sort of reasoning to be a conjunction fallacy causing an epistemic closure.
JoshCole is doing a computation to arrive at the conclusion that there's a <1% chance that this code was added after Musk bought Twitter. I'm using the same methodology with at least equally plausible inputs to arrive at there being a >>99% chance of it.
How is that a nitpick? They're diametrically opposite results.
I'm JoshCole and I didn't find your reply to be a nitpick; you are right that the probability ought to be higher than 1%. My calculation was simplistic and I felt it was prudent to arrive at low probability, because I think probability of wanting something given claim of wanting it removed should probably not be anywhere near close to certain. My estimate isn't 1% though. It was just a short thing to share that gave an intuition for why it might be reasonable to assume he didn't know or want it.
In my opinion if you really care about this topic the right thing to do is ask someone at Twitter when the change was made. Getting more information would make us converge on the true estimate faster than arguing the odds IMO. Feel free to update me with the results if you do end up doing that so I can adjust my beliefs accordingly. I'm not going to try to gain this information, because I don't think the question matters much.
Of course he did because it makes him look bad and he's desperate for praise and attention.
What he wanted was everything that feature provides, without it ever being shown that it's there. But since he refuses to hire PR people and almost certainly came up with this idea in the last few days, no one was paid to hide its existence.
The next story out of Twitter will be the remaining engineers being threatened because Musk can't see his tweet statistics any more.
I read the code snippet before I saw the link and thought you were joking, but yeah, there really is an author_is_elon flag right there in the main branch.
Those aren't redundant or collinear though? Maybe you are surprised they didn't encode this as an integer "num_images"? It is fairly common to one hot encode ordinal variables with only a few common/possible values this way.
True, it still seems odd to encode an explicitly ordinal variable as categorical (particularly one with a small finite range, in contrast to the follower logarithmic bucket ones), but Twitter's layout is weird enough that it could be a impactful difference in terms of engagement.
This is (weirdly) common in production ML codebases written by software engineers. Like you, I have no idea why unless it's a memory optimisation (where you count 4+ as many).
Having every column as a boolean (0/1) means you can treat it as a bitmap. As an (entirely fictional) example, imagine if you wanted to get the features of a thread instead of a single tweet. You could do it as a union of all the tweets:
Ok that makes lots of sense from an engineering perspective. It's pretty insane from a statistical perspective though, which I think was the original point.
Per Zoe:
The code that was released today doesn't show the parts that actually alter the scores of Elon and other users. The part of the code referenced below just tracks Elon stats (from what I know). Employees removed most PII before the code was released.
Probably exactly the most boring, mundane way you would expect: they managed to contact the owner of what was probably a dead/inactive account and then paid them some money for it.
It's against the Twitter terms to pay money for accounts. I think more likely is that Twitter 'seized' the dead account and just granted it to her. I'm sure for instance someone had previously squatted on accounts like "VP" or "England"
Could be that those in charge of preparing this open sourced repository did it begrudgingly, and so they perceived the fact that it looked bad as a positive thing. "Hey, you wanted us to release the code. Happy now?"
Why are you assuming this knowledge is harmful to them? What do you think it means for their business?
No other social media platform will have this sort of accountability and public pressure to be better like having their recommendation algorithms public.
How hard would it be to replace this entire algorithm with the following pseudocode?
If !user.follows_author(author) then don't show tweet on timeline
Else if tweet.timestamp is later than all other tweets show tweet first
This is vastly superior to any other possible recommendation algorithm because users can choose what tweets they see/don't see by whom they follow and everybody has an equal chance to have their tweets seen by their followers. When Twitter moved away from this, it rendered my timeline useless so I started just pulling up people's profiles to read their tweets in order and eventually deleted my (pseudonymous) account that had several thousand followers. Almost nobody was seeing my tweets anyway thanks to this algorithm and deleting the account did not prevent me from browsing accounts I'm interested in.
All Elon needed to do to fix Twitter was to reverse all of the bad changes they've made since 2015 or so and restore the platform to what it was in the late 00s/early 10s.
If anybody is actually reading this thread, it looks like twitter is using "author_is_democrat" and "author_is_republican" to evaluate "Community Notes."
As with all of the media outlets that elevate these two private clubs into the arbiters of truth, votes for Community Notes have to be relatively balanced between the two parties. Bipartisanship is a trash metric for determining truth, but absolutely none of the people raging at Musk in this thread would disagree with it.
The “author_is_elon” flag may have been assigned to him because Elon’s Twitter account has the most followers on the platform.
So, for technical / performance reasons, changes to the algos might want to be benchmarked against this account in particular, because it’s the account most likely to be at the centre of capacity- / load-related issues.
private val DarkRequestAnnotation = "clnt/has_dark_request"
private val Democrats = "democrats"
private val Republicans = "republicans"
private val Elon = "elon"
private val Vits = "vits"
Wait, am I missing something here or author name is cleary mentioned as elon here while musk’s twitter id it @elonmusk? Why is everyone assuming this code is about elon?
I mean yeah it must be elon we know but what I was mostly curious about was if it’s actually meant for him only, why didn’t they use his twitter handle? And just elon? I am not a developer I must mention it looks like. And I was genuinely curious.
"author_is_elon" (notice the _ ) is a name the developer writing that code came up with for a category of twitter users. it has nothing to do with twitter handles. he could have chosen any name, but since that category includes only elon musk, he named it like that.
It has been my personal experience, over 25 years in the industry, that often times the bigger the company the worse the code.
It's not an absolute rule, I've certainly inherited projects in a consulting capacity that were written by small teams and were atrocious. But more often than not, a small team working for a small company has fewer of the internal "forces" that incur "technical debt."
Those forces are things like
- Silo'd teams working on a common code base in parallel but never talking to each other, thus duplicating code and having wildly different conventions
- Layers of middle management each with different management styles, leading to inconsistency and product-wide short-cuts
- Dealing with sudden success-induced scalability disasters that result in bandaid solutions
- More employee churn which means that the way we did things yesterday is not the way we're doing things today because someone new is in charge ... more inconsistency in code and software decisions
- More "old code." Companies very rarely do rewrites and when they do they're often failures. So the bigger the company, the more "legacy" spaghetti code typically because you don't fix what isn't broken (especially when the entire system is broken because it's one big giant mess that no one understands and yet somehow it actually works ... as long as we don't breathe on it or get a sudden surge of new account sign-ups).
Yep. I work for Big Tech, and our CI was lacking for a very long time. Integration testing isn't prioritized at all. Ironic to write code in an overly pedantic language with type safety, spend 3X that time on unit tests with near-100% coverage and what-if analysis, and still have the thing just not work in the end.
For my own projects, I went TDD with integration tests first, and it paid off in both the short and long term. Soon after activated CI (thanks Heroku Pipelines). Actual functionality was more complex than what my day job project does, and my way of doing things wouldn't have been allowed there, yet it was more stable in the end. Amazing what you can do when nobody is telling you what to do.
I think the impressive thing would've been you being able to achieve that while coordinating with multiple other people. Anybody can write "perfect" code with enough reading/practice. I think being able to fit your code within the context of a codebase is another skillset entirely and presents entirely different challenges regardless of management influence
One time I used that approach was in a small startup. I was working with frontend contractors, so I collaborated with them on FRs, API design, and bug reports, but it was mostly separate coding-wise. TDD and docs mattered a lot due to the 12hr time diff and language barrier. The only other backend dev came at the end, and I handed the whole thing off to him. Aside from that, business people.
I only write code that's just good enough. My goal was to quickly handle the business requirements that were a bit of a moving target and keep prod stable, and I did. If we had more people, a few parts of it would've been handed off as services that work similarly to my piece. And I know they would've been fine because my system was nothing special or pretentious, just your typical NodeJS/Postgres backends plus a few other pieces. The frontend devs understood it well enough to tweak new features occasionally if I was asleep.
Big Tech(tm) team would've been fine with this too. We have roughly the same-size system and resources I did, and similar relations with internal customers, except 5 backend SWEs instead of 1, and except we're forced to do things the hard way. FRs take forever, and the system is flaky.
That's definitely what this is. Not a twitter employee but probably all internal projects have a ci.sh that runs on their internal CI infra and they just didn't feel like going through open source review for it.
Joking aside, and assuming it got banked out for security reasons, there is something nice about having CI be a single shell script rather than the proprietary yaml format of your favourite CI provider.
Absolutely. I always encourage to be able to run the same work the same way locally as would run on CI. Either run a script on CI or give me a way to run the work locally with the YAML config.
Well for such a flawless CI setup I would expect a much longer commit history of “trying it again…” and “descending into madness…” and “might as well summon the eldritch…” and “oh no what have I done, …” and “there this should probably work, nothing unspeakable to see here!” and “oh no not again…”
Maybe it's weird, but for all the work I have ever done, I have never used CI/CD in the way that it was meant to be used, or never really leveraged it. Maybe all of my past jobs were unprofessional, but like, I see a lot of jobs using "CI/CD experience required" and I think... huh I wonder if they actually do it
I wouldn't say it's necessarily weird, and I'd never call it unprofessional, but I have also been using CI of some shape or form since I entered the industry in the '00s. From home brewed scripts that were cobbled together internally, to CI servers on-prem, to CI servers in the cloud, and now back to on-prem. At work, I am literally in the middle of a massive migration of my teams multiple CI servers. We have dozens, sometimes hundreds of jobs kicked off on a daily basis + at least 2 dozen nightlies. Without CI, our team would be dead in the water.
maybe I just get hung up on the "professional" aspect. I'd say for the most part I've always had some form of testing, just nothing that ever fit neatly into something like gitlab or github's CI/CD products.
Wait so you just merge right into prod? That sounds scary. The thing I work on runs like tens of thousands of tests on large PRs. It's reassuring but sometimes annoyingly slow.
I am not sure about that. Twitter has open sourced a lot of stuff in the past. There were certainly people there who would run the site as a nonprofit public service if they had the choice.
On the other hand, I kinda loathe Bootstrap. I started web dev a bit after it peaked so maybe that's why but I'm so sick of the look and find it hard to think people ever thought it was good.
Another trend Twitter popularized that I can't wait to die: absurdly large border-radiuses
This release seems less immediately valuable than their other contributions, but historically more significant. It’s a pity we don’t have commits although that would be a huge privacy issue.
But yeah, while I would never work for Elon I’m glad he did this.
its literally led to the complete change of my future due to the ability to follow the interactions of successful people who are active on the platform. I've learned from them as if they were my direct mentors and made huge life decisions based on some of their talking points / motivational mindset. Without it, my life would've been a bubble in Virginia with my nearest network being 5 friends who love cranking out bottle on the weekends.
Politicans and companies all over the world are using it. Controlling that information space, is real power. And I am not yet clear, how much that release will bring needed transparency. As the algorithm in production, can have major tweaks.
The importance of Twitter was it being the primary posting location for a lot of things. A number of the artists and other creatives I know have gotten absolutely gutted by this.
... did you see the source code? Blue checkmarks are weighted 4x as much as non-checkmarks. That's just one example of a major change that wouldn't have happened were it not for Elon
"given I don't think you can run this code there isn't a meaningful way to contribute."
Hm, there is lots of code released, I would think that some of it, can be run and might be forked and useful in other context, but mainly it is a PR move, sure.
People noticing stuff is free work but it’s still different than getting PRs and actual solutions like a real OSS project. And that’s fine, people are doing it because they care or like the attention or outrage and all the other personal/social motivations.
Friday is traditionally end of the week and many people stressed out, tired, frustrated about the week and so on. So fridays the conversations are less nice online I noticed. So apparently many people used their friday frustration to blow off steam by spamming the github page with "issues" like, "rewrite everything in rust", "rewrite everything in scratch" ,... and while I was checking, many people were participating in this.
Elon is addressing this in the Twitter Space right now. "It definitely shouldn't be dividing people into Republican and Democrats; that makes no sense[...] you've identified something we should be getting rid of right away."
So false equivalence is written into the platform. Insane opinions of one party must be displayed as often as moderate opinions of the other. It definitely works for angering everyone on Twitter, not so much for actual dialog or progress.
The amount of stretches people make in these threads is amusing. There's enough real things to care about, don't need to set completely unrealistic standards you'd never impose on yourself or 99% of others you didn't already despise.
IIRC it was very recent, there was a Twitter engineer that was fired after explaining to Elon that the algorithm was not biased against him: https://www.salon.com/2023/02/10/petulant-elon-musk-fired-tw... Almost certainly after that event Elon had them explicitly bump his tweets in their reach.
That's one possible explanation. If you were implementing that change, would you do it by embedding an "elon" variable in the codebase, or would you make the specific account a parameter?
And Elon later explained that that had nothing to do from him making any specific request that his tweets be valued.
This is largely a false creation without evidence to back it up. People are assuming intention behind something that's much more easily explained as a bug, one that was quickly fixed. If it wasn't a bug, why would it disappear if people assume Elon is narcissistic as people claim? It'd still be prioritizing his tweets if it wasn't a bug.
It's an incredible coincidence that, out of hundreds of millions of accounts, this bug happened to elevate the account of the person who owns twitter. I'm not saying it was 100% intentional, but it does suggest Musk's account is 'special' in some way.
Did anyone look to see if anyone else was also elevated? Surely there's an extreme measurement bias if the highest followed account gets boosted. There's no evidence to support the idea that only Elon Musk was elevated.
\*
\* These author ID lists are used purely for metrics collection. We track how often we are
\* serving Tweets from these authors and how often their tweets are being impressed by users.
\* This helps us validate in our A/B experimentation platform that we do not ship changes
\* that negatively impacts one group over others.
\*
So now engineers working on the algo can ensure their launches won't lower Elon's tweet visibility. Looks like those remaining at Twitter have a knack for corporate survival.
Companies can't revoke visas of employees... That's just not how H1B Visas work. Yes they can be fired, but they can go find another job and keep the Visa.
Can they? I don't know, but I imagine that at this point, everyone still working for Twitter is there because they don't have any other realistic option.
Most people working there are looking for other jobs. It is not hard to find another job, but in this market it is hard to find one that pays the same. A lot of people have 4 year stock grants and annual refreshers, pushing their compensation very high which can only be matched at other tech companies, which currently are not hiring much.
That's assuming too much in my opinion. Not everyone has the same or similar opinions regarding their work or Elon, even if it's hard for you to believe.
It's harder with visas, because you're not gambling if you can find another job, you're gambling if your employer can acquire a visa in time for you to not be deported.
Right, so it's still justified to feel bad for visa workers in this case, because they are stuck at an abusive workplace because of circumstances of their visa.
With forced RTO and "hardcore" mandates, it's difficult to find 6+ hours of time to interview at other companies (assuming there are any open visa sponsorships available).
According to this one link [0], H1B "tranfers" are not subject to visa caps. Plus you can search for new jobs and start the process while still employed.
Still smells to high heaven to me. Not the Elon part, I don't really care about that. But collecting metrics about "republican" vs "democrat" sounds like a particularly bad set of priorities at work.
Anyone that has worked in social network recommendations (raises hand) knows that they'll be accused of being politically biased, particularly if the recommendations aren't explicitly promoting biased news sites on the Right. (e.g. The Associated Press[0] is leftist propaganda! Where's unbiased news like Gateway Pundit[1] or InfoWars[5]?!) So data scientists and engineers will get pulled in to investigate the latest ref working[2][4], and this will let them easily determine that no, there is no bias.
None of this will matter though, because the complaints are made in bad faith.[2][3]
You may say this is biased comment, but I’m not going to engage in false equivalences, when the outrage and results of the outrage aren’t symmetrical. Cite one story where a major social network (Twitter, Facebook, Google News, YouTube, etc) publicly came out and said that they were adjusting their algorithms to make it more lefty. I’ll wait. This bad faith of the complaints are particularly obvious when the most popular and influential right wing television channel, Fox News, has been caught red handed knowingly spreading conspiracy theories for ratings.[6]
[1] "The Gateway Pundit (TGP) is an American far-right fake news website. The website is known for publishing falsehoods, hoaxes, and conspiracy theories." https://en.wikipedia.org/wiki/The_Gateway_Pundit
[4] "There is some strategy to it [bashing the ‘liberal’ media]. If you watch any great coach, what they try to do is ‘work the refs.’ Maybe the ref will cut you a little slack on the next one." -- Rich Bond, 1992 Republican Party Chairman https://www.americanprogress.org/article/think-again-working...
That doesn't say they did anything about it. Just that right wing complaints had no substance.
In fact, in just over a year from the publication of that blog post, Twitter -- as a matter of official company policy -- would be promoting the unfounded belief that Twitter engineers and scientists were actively engaging in a propaganda campaign against conservatives.
The exact opposite reaction of what I'm looking for.
I know there's a joke about this regarding his ego and there's certainly some truth in that, however it's also quite believable that after a deployment he might have noticed the popularity of his tweets going down (since he no doubt checks his reach), so I can kind of understand how he might see "republicans", "democrats" and "celebrities_it_makes_sense_to_check_this_with_my_account_as_i_am_a_very_active_user" as core categories that need to have their reach balanced.
> it's also quite believable that after a deployment he might have noticed the popularity of his tweets going down
He did notice it and it was treated as a 5 alarm fire, with a Musk cousin sending 2 am slack messages (on a Monday!) to Twitter engineers to urgently fix Elon's reach[1].
Honestly, if you read behind the lines, it sounds like the employee was intentionally making a joke about it at his expense in front of a bunch of people, and I think a lot of CEOs would take that badly as this is effectively the same thing as calling your boss egocentric.
But, we do have a bit of code that measures metrics on his account, so can we find the bit of code that increases the engagement on his account?
> “When you’re asked a question, you run it through your head and say ‘what is the least fireable response I can have to this right now?’” one employee explained.
Reading between the lines, Musk sounds like a giant baby.
No, the "weights" of your model are trained from the input data.
What is usually AB tested are hyperparameters of the model, or different "flavors" of (model+input data).
What people are implying is still unsubstantiated though. The engineers on the Twitter Space say that this is to ensure that changes they make do not bias one category over another, they don't say that it's in order that they can make discretionary updates to bias towards Elon Musk.
Maybe after every update to the model, they check these stats to ensure that they haven't biased towards Elon Musk, and if so roll the change back.
I would not be surprised if “author_is_elon” was added after he bought the company and worked the engineers too hard to figure out why his tweets don’t have a lot of engagement.
> "Today, the For You timeline consists of 50% In-Network Tweets and 50% Out-of-Network Tweets on average, though this may vary from user to user." I have spent significant effort creating a network and there you go choosing to ignore my efforts by putting in 50% of crap-I-don't-want-to-see. That is why I despise your algorithm.
This is just one feed (the "For You" recommendations feed), they also have the "following" feed tab next to it that is 100% your network (want you want), and it remembers your selection when you change between them (they fixed that a few months ago), so really this is kind of a pointless thing to despise for that reason. It's just an option you can 100% avoid if you don't want to see it.
In fact, Twitter is probably one of the only few left in the large social media space that actually gives you an 100% following network feed (minus maybe ads) in chronological order that REMEMBERS your selection (Facebook, Instagram, and TikTok don't). Which makes this even more silly to say. Facebook, Instagram, and TikTok do all have in-network exclusive chronological order feeds, BUT they are extremely hard to find, or don't remember your selection to them.
Hate of Twitter is easy to spoon out, but at least complain about things that aren't already solved for you.
I'm not on twitter enough for the chronological feed to be appealing to me, and instead want to see the notable tweets from the accounts I follow since the time I last visited. There's no straightforward way to achieve this, but if anyone else has this preference, the workaround is to create a twitter list with all the accounts you follow and set it to show top tweets first.
Is your app up to date? I have it on my iPhone, iPad, and an Android device which all have no problem always remembering the "Following" tab selection. As well as desktop/web.
Says the one who cared to comment in the first place, 16 days ago and you getting angry after taking a simple joke too seriously after the fact that Twitter open sourced it's algorithm.
HNer's like you just make me laugh all the time with silly replies like that just being unable to cope.
I said I despised the algorithm, I did not say I hated Twitter. Now I at least know why I hate it.
Yes "Following" is what I use. The reason I use it is because of this algorithm that thinks I could possibly want 50% tweets that make me "engaged^H^H^H^H^Hraged". To me, that is a ridiculous mixture.
I'm happy they have a "Following" and I sure hope they keep it, but I will not be surprised if it goes away.
Because the Followed feed is purely chronological. An interesting tweet from someone I follow could have happened 3 hours before I opened the app, and I would miss it.
This is why I preferred the old For You tab - it was (mostly) the people I had chosen to follow, but meant that I had the best content show up whatever time of day I opened the app. This is particularly important when I'm in the UK and most of the people I follow are in the US, so they're not tweeting generally at the same time I'm on the app.
On the android app at least, a recent update made it so pressing the home button while you're in the "followed" tab switches to the "for you" tab. It's extremely annoying
I'm not opposed to social media feeds having complex recommendation algorithms. I just wish they allowed you to opt in to a reverse chronological feed of only people you follow, like RSS.
Twitter has this now. The home page is split into two tabs: "For you", the algorithmic feed, and "Following", the reverse chronological feed of just who you follow.
Twitter has always had "chronological timeline" behind a confusing "sparkle" button (except for a brief period a few months back where they removed it, or always defaulted to back to algo timeline? and then restored it a week later)
Not having to login to Twitter is a great feature. I don't have a Twitter account, but often see posts linked elsewhere, and hated getting hit to login each time.
How? By having different requirements and implementing another solution? Look at the guy behind YouTube purge tool. One guy accomplished more than YouTube have done for years. Why? No one knows but it's definitely not because lack of resources. I've definitely seen less spam during the last months.
No, he's talking about the very recent spam explosion in the YT comments, and how a dude manged to build an actual counter to that when YT hasn't really done anything on that front. I guess spam decreased since the peak of 3 months ago, but it's still a massive problem. And again, this is in the past few months, and has nothing to do with the improvements to the yt comment algo changes of the past few years.
That wasn't the same, clicking the sparkle button still included a lot of recommended tweets. The only effect was that it made the timeline chronological and included all tweets from people you follow.
On my “following” tab (on the phone app) , I’m still getting recommendations for bomb throwers I don’t follow. Am I weird? It’s like an unhinged relative. Not pleasant.
Edit: I reversed “for you” and “following” in my original reply.
I find these notifications so confusing - at first glance they look like DMs or mentions to me. I don't follow these people, nor were they RT by anybody I know.
It's not. I follow a fairly small # of people (~500) and getting people to reliably show up in is a long-running problem. Following is not enough, you have to favorite or somehow interact with them sufficiently to be sure of seeing all their tweets. It's quite annoying.
Very sure. I started noticing the issue a couple of years ago (well before Musk's arrival) because I'd find myself thinking 'I haven't seen anything from ____ in a while, I should follow that person', only to discover that I was already following them and their tweets were just not showing up. New follows will generally pop up reliably, but if someone has fallen out of your regular feed you have to work to put them back into it.
Just now (as I am writing this comment) went to check on someone I saw an interview with the other day, and sure enough I am still following the person and they have been tweeting a few times a day, but I haven't been seeing any of it.
Other weird things are Twitter's habit of just preemptively muting people (I'll sometimes wonder why a person didn't reply and go back to reread a conversation, only to discover that they did reply; and conversely, people that I have muted or blocked showing up in my search results for a trending topic. Most of the people I manually mute are 'influencers' who use software, staff, or pure obsession to get in the first reply to politicians and the like, a behavior I find insufferably annoying even if I agree with their position.
I'm very interested in politics, but almost all my mutes/blocks are people of somewhat-similar political persuasion that Twitter assumes I would want to see, and insists on showing me despite my best efforts. I want to keep tabs on the arguments of people I strenuously disagree with, because I already know my own opinions and don't need validation. It's easier in some respects to maintain a second account with an uber-conservative persona and let the recommendation engine just feed it with more of the same.
That was the great thing with the 3rd party client, I could trust that all the people I followed I would get their actual tweets. Every single one of them. There was no also messing with it, no tweet "liked" by someone else, etc. Who I followed is what I saw, nothing less, nothing more.
Of course Elon banned those apps, so now I am on Mastodon where I see 100% of the content that I want. Bonus is that I can even follow many twitter users, through Mastodon bot mirrors. And of course, no ads.
It seems the only reason why people wouldn't show up then is that they got shadow banned or at least have some deboosting applied. So the algorithm thinks they are spam?
Edit: Why am I downvoted? It literally did, it even was named as you'd expect it ("sort by latest" or something), tho the location was less obvious as it was under the stars icon above the feed.
> Twitter has several Candidate Sources that we use to retrieve recent and relevant Tweets for a user. For each request, we attempt to extract the best 1500 Tweets from a pool of hundreds of millions through these sources. We find candidates from people you follow (In-Network) and from people you don’t follow (Out-of-Network).
> Today, the For You timeline consists of 50% In-Network Tweets and 50% Out-of-Network Tweets on average, though this may vary from user to user.
It would’ve been interesting to see what changes were made since Musk’s takeover. As someone who followed 5,000+ users, I know I never saw a tweet that wasn’t either from nor retweeted by someone I followed — e.g. I never saw those “[user you follow] liked [someone you don’t follow] tweet”
50%/50% in FYP seems to reflect my experience today — which is much worse, to the point that I’ll regularly switch to viewing by List b/c I miss seeing people who I want to read.
I wonder how much testing and analysis went into deciding on the 50/50 ratio — e.g. how does it impact user engagement and behavior. Because it sounds like an easy round value that you’d land on when thinking “users should be pushed out of their bubbles”
A year ago my account with 5700 followers got an average of 3000 impressions per post (art). Today it's only 200-500. It mentions their fanout system was replaced by something new, not sure when or if thats in the drop, but my impression count dropped around April-May last year. Clearly something decided my posts should not shown to my followers very often.
Perhaps if you did follow so many people they got drowned out, but with substantially fewer following, those recommended tweets were a big part of what I saw. Especially in the last year or so before Musk took over: Twitter went a lot more aggressive and didn't just show tweets which people you follow "liked", but also other tweets, which the algorithm somehow determined you might like, which was often wrong, and, moreover, so frequent that it made a big portion of the timeline. The "following" tab fixed this problem.
Yep, having had created a few throwaway accounts I definitely got a sense of how the algorithm compensated for the majority of users who aren't super active. And it makes sense -- most new users aren't going to want to spend account creation picking 50 accounts to follow.
But if someone has hit the follow button 1,000+ times, it's reasonable to have some faith that they've seen a lot of tweets and know what they want. Showing a few out-of-network tweets seems reasonable (I got enough as it is through followings' retweets). But 50% of a feed that already can't fit tweets from thousands of followings just feels like shit.
The worst part is that the share of in-network tweets seems to be highly concentrated to the last 10 or so people I most recently interacted with, e.g. seeing the same user over and over just because I liked one of their tweets the other day. Which makes sense to save on computation costs, but it's pushed me into a much tighter bubble than I ever had when the timeline wasn't so out-of-network focused.
The Following tab limits to my followers, but in reverse chrono order. This is also not desirable b/c it limits to whoever’s posting whenever I’ve happened to check the feed.
I’d like to see interesting tweets from a few hours ago, and not just Australian tweets when I’m up late at night.
For ranking the candidates these predictions are combined into a score by
weighting them:
"recap.engagement.is_favorited": 0.5
"recap.engagement.is_good_clicked_convo_desc_favorited_or_replied": 11* (the
maximum prediction from these two "good click" features is used and weighted by
11, the other prediction is ignored).
"recap.engagement.is_good_clicked_convo_desc_v2": 11*
"recap.engagement.is_negative_feedback_v2": -74
"recap.engagement.is_profile_clicked_and_profile_engaged": 12
"recap.engagement.is_replied": 27
"recap.engagement.is_replied_reply_engaged_by_author": 75
"recap.engagement.is_report_tweet_clicked": -369
"recap.engagement.is_retweeted": 1 "recap.engagement.is_video_playback_50": 0.005
Having worked at similar companies on similar systems usually A/B experiments and smaller probability of an action bigger weight it must have to matter much overall. The constants are generally done through some ab tests to get them into reasonable overall behavior but they are a pain to tune and very unlikely optimal in any real sense as it’s often too difficult to do extensive search of them. Like often I’ll see new target have a couple different weights tried on an ab and then maybe second set of experiments after rough magnitude is determined.
> Twitter use to rank posts higher for those who had more followers/less people they follow
> They are removing that as of today but kinda interesting that someone with 10k/10k followers would get less reach than if they had 10k followers and only followed 6k
Might be a language barrier from my side but it doesn't really sound like a "strange quirk" but rather a wise decision. Following people to get followers is one of the oldest spam methods on social media. It's not surprising that they would reduce the reach of accounts using those methods.
And I appreciate it. I'm looking forward to people analysing all of it so we eventually can get an in-depth lecture on both the techniques and their decisions.
I stopped posting because I realised I might get caught in some spam filter, but I think I'll now look some more, there are so many random tidbits you can find in a company's code, it really reflects a company's culture and journey sometimes.
> Twitter is also using the page rank algo that google created. Basically, if a lot of people interact with the user they create more authority in the system.
> Twitter algo is finally opensource.
> • Twitter Blue 2x boosts
> • Likes have 30x comment value
> • Links/mentions/names deboosts
> • Retweets have 20x comment value
> • Restrictions/suspensions deboost
> • Images/videos/trending topics 2x boost
> Will write a thread about it later. GM
This is pretty limited. I picked a term used in the diagram to see what I could find out about it. But there seems to be next to nothing in the released code about the mentioned "author diversity". No real code or description.
Wouldn't any such system depend on 10 other internal systems, 20 databases directly or indirectly, each affecting the behaviour of the recommendation engine. That makes me doubtful studying such a recommendation engine is any better than a purely academic exercise.
In realgraph you can see some of the things they keep track of, which include what you have in your address book, total time spent "dwelling" and a few other interesting nuggets.
While I would never install a platform app because I know what kinds of privacy controls some platforms have - seizing a graph of your phone, sms and email contacts (realgraph) to weight engagement is pretty egregious.
The minority of people who understood what this was already worked for platform companies and wanted to again, and the few who didn't but also knew how invasive this was could always be discredited as conspiracy theorists.
Ever wonder who else gets those graphs from platform companies? Today this is all interesting, but a couple of weeks from now when this all sinks in, I wouldn't be surprised if I were mad as hell.
Since this is what most people are going to want to see:
> We also took additional steps to ensure that user safety and privacy would be protected, including our decision not to release training data or model weights associated with the Twitter algorithm at this point.
While open sourcing code is always great, and kudos on them for doing so, let's be real most people didn't care about the internal plumbing of how their recommendation system runs. It's going to be a mess of decades old code, microservices and ML pipelines just like one would expect. If you want to dig deeper to check for biases (the reason they claimed to be open sourcing it in the first place), you will however run into:
> We also took additional steps to ensure that user safety and privacy would be protected, including our decision not to release training data or model weights associated with the Twitter algorithm at this point.
Democrats / Republicans looks like it was added outside of SDLC [1]. This order without those features is sorted, likely by a linter, suggesting Elon and Vits are properly implemented, and Democrats/Republicans was just inserted alongside the Elon feature, perhaps just for this extract. Sorting it now results in a different order than the commit.
That's my thought as well. Complicated system like this rely on all sorts of related services and data stores. This seems like the sort of thing that sounds a lot more interesting than it is in practice. I would bet many non-technical people expect "The Algorithm" to be a straightforward and self-contained system.
WTF is AuthorIsEligibleForConnectBoostFeature? I guess this may explain why some people seem to accumulate a lot of followers very quickly while all those trying to grow organically seem to struggle. You can imagine if a lot of people benefit from this Connect Boost feature, it would make it impossible for others to be noticed through the noise created by all of these boosted individuals. That's essentially what Twitter feels like ATM. Recently, I manually unfollowed anyone who I suspect may have received a special boost from the algorithms.
It doesn't seem particularly interesting? I would never make a formerly private repo public without first erasing the history. There's no upside to showing everyone your work in progress and almost unlimited downsides.
If they allowed you to git-blame the algorithm, some poor coder would have definitely gotten murdered by a crazy person who thought they purposely changed something to hurt them
So as expected, there is exactly nothing that favors posters from one side of the political spectrum. I don't expect that this article will do anything to calm down those who are convinced otherwise though.
Well written article, from an engineer's perspective.
Ranking is achieved with a ~48M parameter neural network that is continuously trained on Tweet interactions to optimize for positive engagement (e.g. Likes, Retweets, and Replies). This ranking mechanism takes into account thousands of features and outputs ten labels to give each Tweet a score, where each label represents the probability of an engagement. We rank the Tweets from these scores.
This is basically the ultimate black box, so I don't think you can really conclude anything like this either way.
Typically, we expect to be able to run "open source" software ourselves. If you open-source your C compiler, I can compile a C program with it. In a few recent high-profile cases though, companies have "open sourced" ML systems without releasing the model weights. This practice is just like your releasing the builds scripts for your C compiler, but not the compiler itself. While more transparency from social media will be enlightening, calling a release like this (or LLaMA) "open source" feels like equivocation. I'd love to see more full releases, weights included.
Of course --- but without the model parameters, even stubbing those systems would be useless. My point is that while this release gives the public some information about how Twitter ranks tweets, it doesn't tell the story because huge pieces of "the algorithm" are missing. For example: the NSFW classifier "open source" release doesn't tell us anything about what Twitter considers NSFW and what it doesn't.
Not sure if you're being sarcastic, but if you're serious, I'm pretty sure the OP is talking metaphorically. It's just a slight annoyance he's not "emotional" about it.
I also fail to see how someone who is annoyed by code that doesn't follow well established standards is somehow not a good fit in the C-suite.
Did anyone else notice this below? I can’t even begin to imagine how many CPU’s that would require and what the cost must be… just for a recommendation engine.
> The pipeline above runs approximately 5 billion times per day and completes in under 1.5 seconds on average. A single pipeline execution requires 220 seconds of CPU time, nearly 150x the latency you perceive on the app.
I immediately did the same calculation, the climate impact per user also seems non-negligible. Doing some back of the envelope maths, 20W per core server power consumption equals 240.000kWh per hour. At 500g CO2eq/kWh this gives us roughly a billion kilograms of CO2eqs per year. At approx 300M MAUs this is roughly 3.5kg/user/year. Not completely off the charts but still important, reducing the time to 120 CPU seconds per execution would have the similar impact as 300M people not traveling 10-15km by car.
Energy costs per user is also interesting, if at all close, at 0.25c per kWh the power consumption cost per user would be greater than 5$ per year.
Musk said that releasing the algorithm will initially be embarrassing, but that they will quickly update it. So it seems that means they intend to at least regularly publish newer versions.
Of course it could also be that they change their mind when spammers abuse the openness.
Yeah this confused me a lot while reading the comments here. I wonder what percentage of the comments are trolling vs. fell for it vs. think it's legit.
I wouldn't want to use a build system written in Java for non-Java code. Adding the whole JVM as a dependency just for the build system isn't worth it,
In general, I think it's good to limit how much stuff you depend on. Not in an extreme minimalist "write everything from scratch" sort of way, but in a "don't just needlessly add billions of lines of dependency code for the hell of it" sort of way. If you've decided to write a program in Java or another JVM language, you already have some JVM as a dependency, so you might as well use a build system written in it, but when nothing else depends on anything Java-related, I'm gonna need an incredibly good reason to add JVM as a dependency just to be able to use one build system instead of another. And IMO, Meson[1] is good enough, there's nothing I'm sorely missing from it and the build configuration doesn't end up as an unmaintainable mess, so switching it out doesn't seem to cross that threshold.
Then there are some reasons specific to Java itself. For one, the JVM is just incredibly slow to start up, and I hate having to deal with Java-based tooling. Gradle is infuriating to work with for that reason (and others). I'm also incredibly uneasy regarding anything made by Oracle, I definitely don't want to add a critical dependency on an Oracle product just to be able to use a build system which may or may not arguably be slightly better in some areas. I know OpenJDK is a community project, but it's one that's completely dependent on Oracle. With Oracle's recent-ish hostile moves regarding LTS builds of OpenJDK, I'm even more wary than normal.
[1] You may point out that Meson is written in Python, which means using Meson adds a dependency on Python. And yeah, I think that's totally fair, and I would respect someone's decision to use CMake instead of Meson to avoid adding all of Python as a dependency. But Python falls in a different category for me personally, because: 1) a lot of my projects end up with a build-time dependency on Python regardless of build systems, since Python is what I use for things like custom preprocessors and random scripts; 2) the sorts of systems I care about (Linux, macOS) tend to come with Python anyway; and 3) I trust the Python foundation way more than I trust Oracle.
For anything that doesn’t run in “humanly instant” timeframe the startup speed won’t matter - and build tools are definitely in this category.
The fear for Oracle is also completely irrational, Java is probably the safest bet ever — it is not tied to processor architecture, has a specification both on the language and the JVM level (both are uncommon in other platforms), has multiple completely independent implementation and the majority of fortune 500 companies definitely have business critical infrastructure running on top of it, so even if Oracle would do something those can single handedly support the platform indefinitely.
Especially that Oracle has been a surprisingly good steward of the language, and they were the ones that finished open sourcing everything.
> For anything that doesn’t run in “humanly instant” timeframe the startup speed won’t matter - and build tools are definitely in this category.
100% disagree. Waiting multiple seconds as the tool starts up just to get to the point where it actually invokes a compiler is infuriating to me. If you don't have a problem with that, good for you I guess.
> The fear for Oracle is also completely irrational
Again, I 100% disagree. Have you seen how they're treating ZFS? And the lawsuit against Google shows that they consider copying their APIs to be copyright infringement, so I wouldn't bank on non-Oracle-sanctioned community re-implementations.
If you want to tie your C++ code to Oracle, I won't stop you, but I won't be doing that. I'm happy that you found a build tool you like. I will stick with one that's not based on Java.
> 100% disagree. Waiting multiple seconds as the tool starts up just to get to the point where it actually invokes a compiler is infuriating to me
I think Java’s startup time is often overblown, a hello world literally finishes in less than 0.1s, and that won’t significantly grow at the size of a build tool. Plus daemons are a thing. Also, there is not much point in starting the compiler when you don’t even know whether it is necessary.
As for ZFS, what do you mean? Not familiar with the situation, it is open-source and several open source OSs use it without any trouble. It’s just Linus’s overprotective stance against mixing two open-source licenses that makes it unmergable into the kernel, but I do use ZFS on linux every day, so where is the evilness of Oracle?
> I think Java’s startup time is often overblown, a hello world literally finishes in less than 0.1s
The problem isn't hello world, it's the start-up time of big jars. My experience tells me that JVM start-up time on actual large software projects is a real issue in practice. But I haven't used Bazel in particular; I'd be more interested in numbers for that than numbers for hello world.
> As for ZFS, what do you mean? [...] It’s just Linus’s overprotective stance against mixing two open-source licenses that makes it unmergable into the kernel
Haha, that's not right. The problem is that ZFS's license, the CDDL, is (intentionally) incompatible with the GPL, so you can't link CDDL code against GPL code. It's not just an issue with making it part of the main Linux tree either, it's probably not even legal for distros to even ship a ZFS kernel module, since that has to link against Linux's GPL code (according to common interpretations of the licenses; Canonical notably disagrees).
And let me repeat that this mess was intentional on the part of Sun when they made the CDDL (see https://en.wikipedia.org/wiki/Common_Development_and_Distrib...), and it remains intentional on the part of Oracle as they choose to keep ZFS under the CDDL.
The mechanism which prevents Linus from merging ZFS into the kernel, and the mechanism which prevents distros from shipping ZFS out of the box, is the exact same mechanism which prevents Linus from merging any other GPL-incompatible code or distros from shipping any other GPL-incompatible kernel modules.
Nothing is stopping any individual from grabbing the OpenZFS source code and linking it against their Linux kernel; that's legal. Oracle is just doing its best to prevent people from combining OpenZFS and Linux into one package and distributing the result.
From your description I literally only see that “Oracle did nothing at all”, which might be a conscious decision, but it’s not like ZFS would be some secret money grab scheme or something like that.
I'm not saying ZFS is a secret money grab scheme. I'm saying I don't trust Oracle and the way they're hamstringing ZFS is one of many reasons why.
But I think I'll leave the discussion here, it's not about build systems anymore. As I have already said, if your take on Oracle is that they're a trustworthy company and a good steward of crucial parts of your stack, go ahead and use Bazel, I'm not stopping you.
I think the problem is that there is no good cross-platform build tool and as a project grows it almost always depends on some other runtime/language.
I don't want to manage Cargo.toml or requirements.txt or pyproject.toml or package.json or whatever. It's all the same stuff - just a dag of dependencies, each has it's own type and output and build pipeline and defaults and env vars etc.
Bazem, from what I've heard, is the best. Maybe that is the reason for the lack of adoption - it's very good but not good enough?
Bazel is good on many fronts, but it is quite specific to a google-like huge mono-repo structure, where your dependencies are also checked out into the same repo. I believe it can be circumvented, but the base model is that.
No one wants to go to jail for Elon, who has been flagrantly violating FTC orders.[1] There's a good chance the commit history and authors may attest to that.
I haven't read the "algorithm" and this observation might be seriously out of date, but:
for Google Ads, you couldn't easily know what ads would be shown for a given query, without a whole lot of data that's not contained in any code: the experiment settings in the server, for one thing. And the user who's doing the query, for another.
An "experiment" could apply to 100% of the traffic, so it's not really an experiment anymore. And even if you think X has been put into production, there is still a "holdback" experiment, where some part of the traffic does not get X applied to it.
First thing I would like to see gone is business bros sharing 'guides' after you follow them, threatening to start charging real soon. Go fuck yourself, get a real job.
Including the search engine itself in “the algorithm” repo is an interesting choice. Obviously it’s a major player in what gets returned to clients, but the details of that infrastructure aren’t really relevant and is a notable portion of their secret sauce.
Folks talk about media bias: Twitter popularity is a media bias.
It’s the most lazy journalism to be able to write a “news” article about what Kim, or Don, or Elon’s PR team tweeted.
But, as far as “social” this media is: Twitter is a one-way street. There’s no one actually responding or interacting with Tweets. It’s just a comment section to flame bait.
Maybe we’ll all get lucky and Elon will cause Twitter to go away forever.
Unless a trusted third party, forensically audits Twitter, there is no guarantee the published code corresponds to the actual live code in Production. Also multiple parts are not present as stated in the blog.
This should be seen as a possible snapshot of some code, that might have run, might run in the future, or is possibly running in some parts of the production infrastructure at Twitter.
Well that was a giant nothing-burger. This seems to be your standard ranking stack. We find candidates based on who you follow, who they follow, who is trending, and what we think you like. We then rank them based on how likely you are to engage with them and continue to come back and give us money via our subscription service and ad views. We then try to remove spam and other negative experiences.
One flaw I've noticed in Twitter's recommendations recently is the tendency to send notifications for "BREAKING NEWS"-type Tweets. Great, except they're usually for news that happened in the past - typically 12-24 hours ago!
The algorithm really needs to recognise when tweets are time-sensitive and not recommend them just because they got a lot of engagement the previous day!
Heh, I knew it. You need to prune your own following list regularly or become less and less visible. I suspect (but have yet to check) that they also weight visibility in terms of historical follower growth.
That's why you see so many trolls with very low follower counts; it's more effective to make/purchase a new firstname-bunchanumbers account and poop in people's replies than to let Twitter decide placement based on historical factors.
Stuff like this always has consequences, it doesn’t mean it’s a net negative for society. It means you need to adapt and actually fix the problems, while also benefiting more from the accountability.
That’s always been a risk of open source and not being hyper-centralized.
Stuff like this always has consequences, it doesn’t mean it’s a net negative for society. It means you need to adapt and actually fix the problems, while also benefiting more from the accountability.
Given the complex relationship between advertisers, platform, and users I don't know if any meaningful contribution can be made to the algorithm without pissing anyone off. The following tab already gave people who're not interested in algo recommendations a way out. I don't quite understand the reasoning behind open sourcing the algorithm. Any thoughts?
Or perhaps it does nothing at all, and it was there so we talk about it, the "is_democrat"/"is_republican" is also ridiculous, as if the goal was to demonstrate a point about social media in general, hmm
Why do they exist then? No code references it, but that's Scala/JVM so many things depend on runtime initialization, so maybe some other systems do? wich ones?
Is is it there to help fight impersonations? should be solved with Twitter Blue already?
There was reports of people receiving notifications about Musk tweets despite not following him, so..
It's not used at run-time, it's in the repository so that the large language models that are training on the github corpus will know how special elon is, and so that the future code written for twitter by GPT-5 will take the hint and add the favoritism autonomously.
Interesting argument, and definitely worth defending; an AI that's biased by design to remember and preserve the old world order's members rule and influence
Honestly, there's too much garbage in the code dump they made.
Maybe an UML graph or even a presentation or written guide on how they measure and apply each weigh or group policy would make it easier to have some solid take on how it works
it would be super interesting if when logged in to Twitter, you could take a look at your current calculated scores/weights for all the params that are part of these algorithms.
Similar to the Netflix "Stats for nerds" menu...
Will this make it easier to game the algo or does it depend so heavily on individual user interaction that it’s close to impossible to game it? For example, by carefully crafting Tweets or by buying likes/retweets etc?
If Twitter was 'dead' why on earth are we still talking so much about this blue bird site?
It looks like once again these lot predicting that he won't open source the algorithm and are going to start eating their words again [0], just like they did around incorrectly predicting Twitter's immediate collapse [1] and will look at the source code anyway and continue to talk about "Twitter" again.
If Twitter can open-source their algorithm, Why not TikTok? Either way, the bots are now going to have a very expensive time on Twitter.
Are you kidding me, running a botnet is easier than it has been in years if you're that way inclined. The amount of spam I see has gone way up over the last 6 months.
> running a botnet is easier than it has been in years if you're that way inclined.
Even if it is 'easier', the bots are identified, down-ranked straight to the bottom and shadow-banned to invisibility. It is essentially evaporating money and time.
> The amount of spam I see has gone way up over the last 6 months.
Yeah. The spam has gone way up into smoke over the last 6 months. It is only going to get more expensive to spam as soon as the paid changes come in.
So, the day after the headline that Twitter is artificially promoting polarizing political voices, Twitter open-sources their algorithm!
What does the commit history say? There are 3 commits, like a very very real programming project.
The issues and pull requests show how much people are fooled by this very transparent move.
So this is an obvious attempt at a digital potemkin village, that like the real one, poorly succeeds in hiding the truth.
Elon does not not want to upset the apple cart (political economical or ideological) but make his followers believe in it, and so we get this.
Great spectacle, if that's what you're interested in.
The issue tracker and pull requests are being hit with very funny suggestions. Many people suspect this is an April Fools joke. It's possible this entire repo was generated by a LLM to appear plausible.
I especially like the suggestions to rewrite the algorithm in Rust [1] and this pull request which simplifies the algorithm to a single c file [2].
And yet they require their software engineer applicants to be well versed in algorithms and data structures? These tech company managers know nothing about how the sausage is made.
It's a really scary codebase. Do you really need that much code for the world's crappiest recommendation algorithm? I think you can do more crap with less code. we trust you elon.
I couldn't care less about Twitter's high level abstractions. They were never renowned for those. Their database schemas and infrastructure on the other hand...
Is there demand for a service that simply shows you the things the people you follow wrote? (It would be up to you not follow so many people that you can't keep up.)
A plain follow stream is a firehose of mundane messages: everyone you followed's messages, sorted by most recent first.
If some people you follow are more important than others (family) that doesn't matter to the stream, and you get bogged down by less important messages.
I think some "algorithm" is necessary, but people will disagree on the balance. (It's unfortunately in twitter's interest to push all kinds of random shallow stuff and get people addicted to that.) I hope mastodon can maybe provide some flexibility and customizability in terms of what the mix between recent and likely to be interesting should be, and what interesting means to you.
Not that I use twitter much, but since it became clear that Elon made sure to promote himself in the algorithmic feeds, I've avoided "For you" anyway since I don't accept that in my mix of messages.
Because I'm not on Twitter 24 hours a day. I want a recap of the best stuff that's happened since I last opened it. If I use the Following tab, I'm only getting the realtime firehose, meaning if the best stuff happened an hour ago I'd miss it.
To see what people talk about in your friends circles. It can be interesting in moderation. Similar to skimming the frontpage of HN or recommended list on YouTube. Especially during major news event when your friends might not be the ones posting about it.
Probably the same reason some people browse /r/all on Reddit. I think the desire for that sort of thing has waned a lot over the past couple of years, though.
I'm no fan of either Twitter nor Elon Musk, but this is a great move and I hope other companies follow what Twitter did here and start open sourcing more core parts like this. Maybe it's mostly useful for learning how it works, not for directly using it in your own product, but the amount of transparency it gives users cannot be understated. As long as that actually is the code they run, but there would be no way for anyone but Twitter to verify that.
I think it mainly helps with accountability regarding free speech. They did and do several kinds of shadow banning and down-boosting to combat spammers, which always has some false positives. If you the algorithm is published, you could at least better judge and argue when you are unfairly "silenced". Since this may be due to an avoidable flaw of the algorithm instead of some accepted collateral damage.
Astounding amount of cynicism here, so I'll say something positive: Transparency is undoubtly important, I'm glad we can see how all of this works and what sort of effort goes into building a social media system. It's licensed under GPL which is a bummer (would have preferred BSD) but it's better than nothing.
GPL would be good if it is a self contained library. If anyone would use it, it would be small portions of it, but GPL makes it completely useless. You can't contaminate anything with it. We'll stare at it, that's about it.
That makes me think, this is actually a good call. Twitter can claim that they have complete transparency while not allowing anyone to touch their code (because it is GPL). "Anyone" being future competitors. If it was BSD licensed, it'd be tremendously useful in building a Twitter competitor (on paper, you still need network effects, I am just spitballing to make a point).
Summary this far
• Code from Twitter's algorithm GitHub repository shared
• Algorithm checks for specific author types (e.g., Elon Musk, power users, Democrats, Republicans)
• Author ID lists used for metrics collection in A/B experimentation platform
• Metrics tracked in A/B tests to avoid negative impacts on specific groups
• VIPs like Musk, LeBron James, AOC used as indicators for algorithm's behavior
• Algorithm changes that negatively affect Musk unlikely to go live
• Speculation about code changes pre- and post-Elon's purchase of Twitter
• Discussion on the importance of measuring and testing for potential biases
• Debate on moral decisions in the context of Twitter's algorithm and content moderation
Late to the party here so unlikely anyone sees this comment. But the double take for me was seeing the article end with "if this sounds interesting to you, come join us!"
if task == 2:
if language == "ar":
self.query_settings["adhoc_v2"]["table"] = "..."
elif language == "tr":
self.query_settings["adhoc_v2"]["table"] = "..."
elif language == "es":
self.query_settings["adhoc_v2"]["table"] = f"..."
else:
self.query_settings["adhoc_v2"]["table"] = "..."
return self.query_settings["adhoc_v2"]
if task == 3:
return self.query_settings["adhoc_v3"]
raise ValueError(f"There are no other tasks than 2 or 3. {task} does not exist.")
Looking through it, the ... seems to be a placeholder for information they'd prefer to be kept private. For example, look in the keywords section in the same file you shared.
I wonder if it will be possible in one day to know what is values of `author_is_power_user`, `author_is_democrat` and `author_is_republican` for your account. Does GDPR help with that? probably not because maybe they do it for people inside the us only so it is not related to EU anyway.
If you look at the GitHub repo, most of it is READMEs describing systems, not the models or code subleties which actually give explanations into how certain weird behaviors on Twitter happen. (e.g. the preference of certain users in the For You tab. EDIT: bad example, since there appears to be a flag for that in the code, although it does not specify which users are on the list)
Where is the file containing accounts that are artificially boosted? We can guess what its single line is, but how is it incorporated into the algorithm?
Or an alternative view: no matter what he does there will always be haters and there are massive (political) incentives to destroy his public persona.
Those incentives also align nicely with those of hedge funds that are short TSLA as well as family offices that are short Tesla (hello Billy boy Gates).
Or, hear me out, maybe he is just a terrible person with some insane level of narcissism who will talk shit about everything without being half competent at that, and recently the consequences of his very own shitty decisions have been catching up to him.
You can be only so stupid to acquire Twitter overpriced and drive it to the ground, there is no need for “political incentives”, when he is quite good at destroying his own persona.
Name 1 Musk company that failed? Incompetent how?! If Musk is considered incompetent, I might as well drop whatever I'm doing forever because I'll never ever even come close?
Driving into the ground? Sounds like pure speculation based on nothing as long as Twitter isn't bankrupt?
You don't like Musk. OK, fine, you're allowed not to like him, but I don't get your reasoning.
You can't argue this alternative view in good faith because he now has such a long record of pulling the wool over people eyes.
Its just mainstream now and so it looks likes its partisan but to be honest people in the legacy auto and space industry have been screaming from the rooftops for years but no one listened to them.
Gets him a positive news cycle without much cost. It's hard to immediately say how interesting or useful this will be since there are probably a number of related systems and databases that aren't released.
It's because there's been nearly a decade of conspiracy theory around the use of algorithmic feeds in social media generally, and Twitter specifically. Among the right, "algorithms" have become symbolic of the machinery of leftist oppression they believe to be arrayed against them by modern media.
So this language is Elon signaling that he's presenting the "woke hivemind's" head on a platter.
I generally have a very low opinion of social media platforms, but I did create a Twitter account for the first time after Musk bought the platform.
My conclusion is that it's basically entertainment, with very little of what I'd call high-quality useful information that deserves further examination (unlike a lot of HN posts, in contrast). I also notice something of a Tik-Tok approach to video being implemented, which is not surprising given Tik-Tok's success (and makes one wonder who exactly it is lobbying so hard for a Tik-Tok ban, and whether it's just a commercial competition issue more than anything else).
As far as the recommendation algorithm, it appears to be a siloing setup - look at content of one particular flavor, it gives you more of that flavor. A 'flush settings' or 'forget browsing history' or 'reset to defaults' button would be useful, if probably not what advertisers want in terms of delivering to target audiences. I suppose setting up multiple accounts is something of a solution, although too much effort to be that interesting.
In terms of news reports, it's broader in scope than traditional corporate media outlets, so that's a plus in its favor. Reliability is perhaps similar (i.e. low).
You can follow accounts that only post arxiv.org links for ML papers or anything else you're interested in if you want to. If you're only getting entertainment then it says a lot about the original accounts you followed.
From a very quick skim of the repositories, this appears to be quite limited transparency. The documentation gives a decent high-level overview of how Tweet recommendation works—no surprises—and the code tracks that roadmap. Those are meaningful positive steps. But the underlying policies and models are almost entirely missing (there are a couple valuable components in [1]). Without those, we can't evaluate the behavior and possible effects of "the algorithm."
[1] https://github.com/twitter/the-algorithm-ml