Hey HN, I posted this in a comment thread the other day and (to my surprise) it got a positive reception so added a few more updates and decided to post it proper.
The idea is to be able to choose a website, select the data you want, and make it available (as JSON, CSV or an API) with as little friction as possible.
Kimono was the gold standard for a while so did yoink some of their ideas, while doing some other things differently.
Still needs some work but as an MVP would appreciate any feedback. Cheers.
When I saw this service last week, I think you had a section about a paid service where you do the scraping on a server and send the results. Do you offer that? How do you get around anti-scraping technology, if it exists?
Yeah, that's offered although it's currently free.
No particular tricks to avoid detection. It's Puppeteer under the hood with a few customizations which works well on the majority of sites tested so far.
Given the cat-and-mouse game around web scraping you may never cover every website, and that's ok.
The idea is to be able to choose a website, select the data you want, and make it available (as JSON, CSV or an API) with as little friction as possible.
Kimono was the gold standard for a while so did yoink some of their ideas, while doing some other things differently.
Still needs some work but as an MVP would appreciate any feedback. Cheers.