I admit I'm one of those people. After decades where I should perhaps be a bit more cynical, from time to time I am still shocked or saddened when I see people do things that benefit themselves over others.
But I kinda like having this attitude and expectation. Makes me feel healthier.
i like that Veritasium vid a lot, i've watched it a couple times. The thing is, there's no way to retaliate against a crawler ignoring robots.txt. IP bans don't work, user agent bans don't work, there's no human to shame on social media ether. If there's no way to retaliate or provide some kind of meaningful negative feedback then the whole thing breaks down. Back to the Veritasium video, if a crawler defects they reap the reward but there's no way for the content provider to defect so the crawler defects 100% of the time and gets 100% of the defection points. I can't remember when i first read the rfp for robots.txt but I do remember finding it strange that it was a "pretty please" request against a crawler that has a financial incentive to crawl as much as it can. Why even go through the effort to type it out?
EDIT: i thought about it for a min, i think in the olden days a crawler crawling every path through a website could yield an inferior search index. So robots.txt gave search engines a hint on what content was valuable to index. The content provider gained because their SEO was better (and cpu util. lower) and the search engine gained because their index was better. So there was an advantage to cooperation then but with crawlers feeding LLMs that isn't the case.
This is a really cool tool. I haven't seen it before. Thank you for sharing it!
On their README.md they state:
> This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.
> Trust by default, also by default, never ignoring suspicious signals.
While I absolutely love the intent of this idea, it quickly falls apart when you're dealing with systems where you only get the signals after you've already lost everything of value.
It's easy to believe, though, and most of us do it every day. For example, our commute to work is marked by the trust that other drivers will cooperate, following the rules, so that we all get to where we are going.
There are varying degrees of this through our lives, where the trust lies not in the fact that people will just follow the rules because they are rules, but because the rules set expectations, allowing everyone to (more or less) know what's going on and decide accordingly. This also makes it easier to single out the people who do not think the rules apply to them so we can avoid trusting them (and, probably, avoid them in general).
In Southern Europe, and countries with similar cultures, we don't obey rules because someone says so, we obey them when we see that is actually reasonable to do so, hence my remark regarding culture as I also experienced living in countries where everyone mostly blindly follow the rules, even if they happen to be nonsense.
Naturally I am talking about cultures where that decision has not been taken away from their citizens.
> I also experienced living in countries where everyone mostly blindly follow the rules, even if they happen to be nonsense.
The problem with that is that most people are not educated enough to judge what makes sense and what doesn’t, and the less educated you are, the more likely you are to believe you know what makes sense when you’re actually wrong. These are exactly the people that should be following the rules blindly, until they actually put in the effort to learn why those rules exist.
I believe there is a difference between education and critical thinking. One may not have a certain level of education, but could exercise a great degree of critical thinking. I think that education can help you understand the context of the problem better. But there are also plenty of people who are not asking the right questions or not asking questions - period - who have lots of education behind them. Ironically, sometimes education is the path that leads to blind trust and lack of challenging the status quo.
> the less educated you are, the more likely you are to believe you know what makes sense
It actually frightens me how true this statement is.
To reinforce my initial position about how important the rules are for setting expectations, I usually use cyclists as an example. Many follow the proposed rules, understanding they are traffic, and right of way is not automagically granted based on the choice of vehicle, having more to do with direction and the flow of said traffic.
But there's always a bad apple, a cyclist who assumes themselves to be exempt from the rules and rides against the flow of traffic, then wonders why they got clipped because a right-turning driver wasn't expecting a vehicle to be coming from the direction traffic is not supposed to come from.
In the end, it's not really about what we drive or how we get around, but whether we are self-aware enough to understand that the rules apply to us, and collectively so. Setting the expectation of what each of our behaviors will be is precisely what creates the safety that comes with following them, and only the dummies seem to be the ones who think they are exempt.
As a French, being passed by the right by Italian drivers on the highway really makes me feel the superiority of Southern Europeans judgment over my puny habit of blindly following rules. Or does it?
But yes, I do the same. I just do not come here to pretend this is virtue.
The rules in France are probably different but passing on the right is legal on Italian highways, in one circumstance: if one keeps driving on the lane on the right and somebody slower happens to be driving on the lane on the left. The rationale is that it normally happens when traffic is packed, so it's ok even if there is little traffic. Everybody keep driving straight and there is no danger.
It's not legal if somebody is following the slower car on the left and steers to the right to pass. However some drivers stick to the left at a speed slower than the limit and if they don't yield what happens is that eventually they get passed on the right.
The two cases have different names. The normal pass is "sorpasso", the other one (passing by not steering) is "superamento", which is odd but they had to find a word for it.
Not sure if it is a virtue, but standing as a pedestrians in an empty street at 3 AM waiting for a traffic light to turn green doesn't make much sense either, it isn't as if a ghost car is coming out of nowhere.
It should be a matter of judgement and not following rules just because.
I kind of agree. The rules for safety should be simple, straightforward, and protect you in the "edge cases", i.e. following while not paying 100% of attention, protect you with a malicious actor in mind aka reckless driver, etc. Ideally, in a system like that it should be a difficult and intentional behavior if one wanted to break the rules rather than to follow them.
I agree. I mostly mean that it is good to strive towards a system of rules that will be easy to follow and difficult to break by default. That is an ideal case. In reality, it is never that simple.
> For example, our commute to work is marked by the trust that other drivers will cooperate, following the rules, so that we all get to where we are going.
That trust comes from the knowledge that it's likely that those drivers also don't want to crash, and would rather prefer to get where they're going.
I apologize for that. I try to mitigate my US-centricness in my comments as much as possible, understanding completely that I am speaking with a global audience, but I am definitely not perfect at it :D
I suppose the same goes if you take the tube, ride a bike, walk, etc? There's still rules in terms of behavior, flow of traffic (even foot traffic), etc, that helps set a number of expectations so everyone can decide and behave accordingly. Happy to hear different thoughts on this!
