To any that have experience getting comments data from HN -- what's the fastest, most polite way to do this? And am I correct in remembering that there's some aggressive rate-limiting for crawling the site?
Any plans to try to get a dataset for supervised ml? Perhaps collect the top 4 comments from all front page threads and post a survey on HN asking HNers to rate those comments for skepticism/dismissiveness?
To any that have experience getting comments data from HN -- what's the fastest, most polite way to do this? And am I correct in remembering that there's some aggressive rate-limiting for crawling the site?