Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
mikemotherwell
on Aug 12, 2020
|
parent
|
context
|
favorite
| on:
Court dismisses Genius lawsuit over lyrics-scrapin...
Other way around. It was invented to stop crawling. Indexing is still technically allowed even when blocked by robots.txt from crawling.
Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
