I used to care about respecting robots.txt until it was clear that established search engines are fine but any newcomers can go right to hell.