Having /includes/ under document root - and trying to fix this via a robots.txt ...

mdasen · on Jan 20, 2009

The includes folder looks like it's just JavaScript and CSS (including jQuery) so it has to be under the document root.

nir · on Jan 20, 2009

Sorry, I'm taking that back... I was thinking of Obama's campaign site where you could actually access stuff like /includes/footer.php (at least for a while, seems to be fixed now)

eli · on Jan 20, 2009

You're making assumptions about what that directory is for. Perhaps it is used to store static HTML pages that are included from IFRAMEs on other pages.

jonursenbach · on Jan 20, 2009

Well that's at least tons better than hiding the entire site with hundreds of Disallow entries, when they could have just done /*.