I have noticed this too. Entire vast websites seem very poorly indexed (I can never find relevant reddit comments, Tweets, etc), and the web feels smaller and smaller, if I judge by how many useful results I get in a search.
I have no idea why that is, but it's really hard to find relevant information for anything outside mainstream stuff. Just the other day I was looking for how to post JSON in a test with Flask, and I hadn't found anything after tens of searches. Surely, something must be referenced in some code on GitHub, or a reddit comment, or some blogpost somewhere, but it proved impossible for me to find.
I agree that there are problems searching large volumes of user-generated content when it isn't referenced much long term (Reddit, Twitter, etc.). If I don't keep record of Tweets myself I rarely find them again.
I have no idea why that is, but it's really hard to find relevant information for anything outside mainstream stuff. Just the other day I was looking for how to post JSON in a test with Flask, and I hadn't found anything after tens of searches. Surely, something must be referenced in some code on GitHub, or a reddit comment, or some blogpost somewhere, but it proved impossible for me to find.