I can't speak to why it was originally defined, but since the Referer [sic] header has existed for decades, many sites depend on it to function. The Smart Referer extension whitelist[1] and bug tracker[2] have several examples.
> I can't speak to why it was originally defined, but since the Referer [sic] header has existed for decade
I can remember my Dad getting a mail from someone he linked to that was about to move his website and politely contacted his neighbors on the internet to allow them to update their links.
It can still be useful for that kind of thing. When I notice an unexpected spike of traffic on one of our sites I'll often look at our analytics to see where it came from and then potentially drop in there to answer comments and such. Not to say that's worth the privacy trade-off though, unfortunately.
Believe it or not, there actually exist websites that rely on the Referer header for navigation. The last time I bumped into this was a few years ago, but a local government site refused to work unless my browser sent that header.
Granted, this is probably rare enough that it's safe to disable the header for the vast majority of websites, but it's something to keep in mind.
Judging solely by the UI, I actually kinda like Atlassian's tools, but they're a huge pain in the ass to get working with privacy extensions installed (uMatrix, uBlock, etc.). They make cross-site requests all over the place (to weird servers like "some-huge-name-that-obscures-the-host-name.atl-pass.net", and even some third party servers!), tons of Javascript and css for basic features, etc. Using dubious features like referer headers seems right up their alley.
It's one of the main reasons I only use them at work, and won't use them for my personal projects. I'd rather pay for GitHub and Sourcehut so I don't feel like I'm opening my browser up to a bunch of security problems.
In the past they've also made some really brain dead (IMO) decisions like going out of the way to break middle-click paste on Linux.
>They make cross-site requests all over the place (to weird servers like "some-huge-name-that-obscures-the-host-name.atl-pass.net", and even some third party servers!), tons of Javascript and css for basic features, etc
If you like this, you should try Microsoft. They combine this crap with endless redirects. Usually, I give up after 5 minutes whitelisting + redirects.
Beyond what other people mentioned, some sites and frameworks also rely on the Referer header as part of CSRF protection. It's not truly necessary to check, but it's an OWASP recommendation so it seems like a decent number of places implemented it by default.
I recently got the Pyramid Python framework to make it possible to disable Referer-checking for the built-in CSRF protection, but they're still going to keep requiring the header by default: https://github.com/Pylons/pyramid/issues/3508
More discussion about it in these pull requests too:
The new version with it being optional hasn't been released yet, so as of right now almost everyone using Pyramid will still require users to send a Referer header to get past any CSRF checks.
I had an old website hosted under www. When it was decided to build a new website, to preserve the old content, the new site was built without a leading subdomain.
The problem was that chrome cached www as the default for anyone who'd visited the old site, and had started hiding www from the address bar.
I used Caddy to redirect all requests to the subdomain free site unless the request came with a referrer from that site, fixing the caching and allowing for free navigation between and within both the old and new site.
