Because everyone "being nice" is how Google keeps its monopoly on search. Googlebot can do anything and everything and no one complains. Or how about sites like Twitter and Instagram that live off selling their user's data having extreme limitations on their public apis and aggressively blocking alternative frontends like nitter or bibliogram because OF COURSE god forbid someone could want to look up something on their platform and not have an account.
The typical response to people raising these issues is "buuuut xy is a private platform that can do what it wants", yes, but why are you defending technocrats with bigger profits than many nation state's GDPs? (Reasonable) crawling should be allowed and promoted, in fact, it should be codified in law as a necessary element for the future of open and free internet. Anyone trying to prevent it, or even worse, make it illegal, is a bad actor.
The typical response to people raising these issues is "buuuut xy is a private platform that can do what it wants", yes, but why are you defending technocrats with bigger profits than many nation state's GDPs? (Reasonable) crawling should be allowed and promoted, in fact, it should be codified in law as a necessary element for the future of open and free internet. Anyone trying to prevent it, or even worse, make it illegal, is a bad actor.