> so I know it's crawled that page, it often doesn't return that page either.
This is your misunderstanding. The fact that a thing was in the index does not ensure it will always be there. Things disappear from the web all the time. Serving fresh docs means not only crawling the new stuff but also deleting the unreachable stuff promptly.
No, I literally did a site: search seconds after visiting the site to see if it had any other pages with what I was looking for, and it found zero results --- not even the original page I found.
This is your misunderstanding. The fact that a thing was in the index does not ensure it will always be there. Things disappear from the web all the time. Serving fresh docs means not only crawling the new stuff but also deleting the unreachable stuff promptly.