Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You dont need to scan the whole dataset. Partitioning + clustering, especially for time-series, should be very efficient. BQ flat-rate pricing starts at 10k/month for a fixed number of processing slots. You can buy more slots to fit your query load.

Anyways the point is that it's well within the financial means of Twitter, especially when compared to developing and operating this proprietary system.



If I'm not mistaken Twitter is mostly on GCP so they could just go with BigQuery instead of developing an in-house solution. We probably need to ask them in order to get a proper answer. :)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: