This is exactly what the list-unsubscribe-post header from RFC8058 provides: https://www.rfc-editor.org/rfc/rfc8058. The unsubscribe button that gmail, Apple Mail and others displays is driven by that; it's not a gmail feature.
Weirdly, if google thinks you're a dodgy sender they won't display the button, which seems counterproductive to me.
Even gmail's own marketing messages (that I never asked for!) end up in my spam folder. If google can't even reliably send emails to themselves I don't know how they expect anyone else to succeed.
One of the key problems is that both gmail and Yahoo UIs actively encourage users to report messages as spam rather than unsubscribing. Yahoo is particularly bad at this; it's common for me to receive spam reports from yahoo on an entirely double-opt-in social site I run. My reaction there is to remove the reporter from all lists because the amount of damage a single spam report can do is immense; a single spam report can block delivery for weeks at a time to the 10k others that legitimately requested messages. Hotmail/outlook/live is much the same in encouraging spam reporting over unsubscribe, however, their penalties are not as excessive as Yahoo's.
> One of the key problems is that both gmail and Yahoo UIs actively encourage users to report messages as spam rather than unsubscribing.
I think this is, generally, the correct approach. There isn't really a salient reason to discriminate between "email I don't want from someone I don't know" ("true spam", if you will), and "email I don't want from someone I do know" (aggressive newsletter campaigns et al). Spam is the button to send a signal that you got an email you didn't want.
> My reaction there is to remove the reporter from all lists because the amount of damage a single spam report can do is immense; a single spam report can block delivery for weeks at a time to the 10k others that legitimately requested messages.
This is the system working as intended to me, as the customer of the email service. I like that my email provider is throwing their weight around to put the fear of God into bulk senders and forcing them to think about how this campaign will impact their sendability. I would much rather annoy the hell out of bulk senders than cede emails to spammers like we have with phones.
There is one case where differentiating makes sense: Sometimes users sign up for newsletters, want the newsletters, would re-confirm if asked... and later change their mind and no longer want those newsletters. Here, marking as spam is unreasonable.
In most other cases (e.g. newsletters sent based on a tiny pre-checked checkbox or without asking for consent), the spam button is of course the right tool.
None of that seems new. What would be new is if both gmail and Yahoo provided any means of allowing legit bulk senders to actually send properly.
The one-click unsubscribe is from 2017's RFC8058. Everyone that's sending in volume is already doing all the usual stuff - DKIM, SPF, DMARC, matching reverse IPs, etc.
The privacy-first email marketing service I wrote (https://info.smartmessages.net) implements account-wide unsubscribe by default (unsubscribing you from one list unsubscribes you from all). It requires double opt-in, and asks for explicit consent before doing any tracking whatsoever (so no Google Analytics, no cookies, no trackers), which of course is what the (at least EU and UK) law requires. You're not going to see shitty exploiters like MailChimp doing anything like this; abusing your data is just too lucrative.
It's still ridiculously hard to deliver messages at any volume, and there is zero recourse when you are penalised incorrectly. Gmail's spam filtering is just dire - if I send myself an email from gmail, it goes into spam. A large proportion of the spam I receive is sent from gmail.
Google's postmaster tools are a joke. It's entirely normal for them to give you a "bad" spam rating when you have 0 spam reports, 0 auth failures, strict DKIM and DMARC, and every single message has double-opt-in audit trails. This useless feedback makes it very difficult for senders to actually comply with their ever stricter, but ever more opaque requirements.
Proving that subscribers actually want to receive messages from you its difficult. So back in 2017 I wrote an outline proposal https://github.com/Smartmessages/subscriptionproofrfc to create a standard, possibly built on top of DKIM keys, to provide provable subscriptions. This would pretty much solve the entire thing for legit senders, but of course the industry is not really interested in cooperation or complying with any law that might reduce the number of people they send to by even the tiniest amount.
Get a grip. So you would consider twitter and Facebook notifications you have explicitly requested as spam? I also run a social network with 20k users that is equally accessible over web and email, and we have had peaks up to about 4 million messages/month containing nothing but conversations between users, and we run into all these bulk sending issues all the time. Please explain, oh wise one, how is emailing people messages they have explicitly asked for spam?
That doesn't help - user input is already validated by default in PHPMailer - the problem is that a valid email address can also be an attack string in a shell context, and bugs in PHP make it hard to work around safely.
That's actually a part of the problem: "don't worry, the library covers you, no need to think about it". I have spent some time rooting around inside the libraries I planned to use - including PHPMailer - to find out what their assumptions are: "validates by default" is not a sufficient description (as we see), "passes everything conformant to RFC" is.
In this case, that was not the same assumption I was making, and I was surprised - once, during implementation; a wrapper using filter_var brought the assumptions of the library in line with the assumptions for the project. That is, obviously, not universal advice.
I wonder why they don't provide a mitigation. It shouldn't be impossible to release a simple sanitizeAgainstCVE_2016_10045($email) function as a mitigation that people could right now plop into their applications until phpmailer gets fixed... or is filter_var($email,FILTER_VALIDATE_EMAIL) good enough?
For benign e-mail addresses, it is Good Enough: accepts the unusual but valid info%blah+something@info.swiss, punycode (and various other things that people usually toss out with the infamous [a-z]{2,4}), yet throws out the abovementioned monstrosity, even though it technically follows the letter of the RFC.
As for "why no mitigation" - even though it abstracts the horror of mail(), a library is a power tool with sharp edges, not a nicely wrapped single-button selfie app. In other words, it already does validation; input sanitization is beyond its scope IMNSHO (this literally means that valid, insane e-mail addresses do exist), and adding that by default would nerf the library. (Also, php_filter is an extension and not installed by default)
This is incorrect. Input sanitisation is not the problem here - PHPMailer sanitises and validates email addresses well enough. The problem is that a valid, sanitised email address can also be an attack in an inappropriate shell context. Much as an SQL injection attack string may be harmless in a shell context but lethal in SQL. Also the vuln does not apply to CLI sendmail, only when sending via the PHP mail() function, and it's at least partly due to bugs in PHP itself.
I am not blaming lack of input sanitization, the bug is clearly still there. What I am saying is that input sanitization would prevent this attack because it will only allow valid characters to get to your mailer, which will exclude spaces, tabs, double and single quotes. Thus I fail to see an exploit vector which would bypass proper input sanitization. Do you see it?
In general I agree that PHP mail() function should take a part of blame here for not exposing a sane interface. Spammers have abused many bugs where programmers failed to sanitize "To:" or "From:" addresses and attackers could pass \r and \n characters (which allowed adding BCC fields, effectively sending e-mails to arbitrary addresses).
> Also the vuln does not apply to CLI sendmail...
Sure it does:
>> ...will result in the followig list of arguments passed to sendmail program:
EDIT: you probably meant to say that the vulnerability is only triggered when CLI sendmail is called via mail(), which is true. Configuring PHPMail to use sendmail via CLI directly apparently doesn't expose this vulnerability.
> What I am saying is that input sanitization would prevent this attack because it will only allow valid characters to get to your mailer,
you have to be careful with the meaning of "valid" though: What's valid in one context might not be in another context. Do you want to limit the character set to the least common denominator of non-special characters valid in all possible contexts your address might be used?
What if you don't exclude some character because it's not special in any of the current contexts but then you later add another thing to the mix that uses in-band signalling? Now you need to update your whitelist and remove even more "invalid" characters.
This won't end well for you.
I would recommend you don't put a restriction on the input (aside of what the RFC defines as valid) and instead correctly escape (or throw if your context doesn't support escaping) when moving the raw data to a new context.
In this case, I think the problem is in PHP's `mail()` function that put stuff in shell context without any escaping.
The fix should be happing in `mail()` (which is internally shelling out and thus switching context), not in the caller and certainly not in the frontend controller when it needs to decide whether a given email address is valid or not.
I think you are taking my argument too far. Valid input doesn't mean it is safe to use in any context. It just means that anything which is invalid at the start shouldn't even have the chance to get in the system.
You should still properly escape data when passed to another context, no doubt about it.
I am also not suggesting "dirty" practices like replacing double quotes (Wordpress) or magically escaping quotes on input (PHP prior to... 5?). But if you know the input should be a number, allow only digits and '.' and clean everything else. And validate the range too! If you need a database ID, validate form and existence. If you need an e-mail address, run it through filter_var. Always. There is no reason not to.
It all sounds very interesting, however, the tone of the web site comes across like some crappy download accelerator. When you follow the "nerdy details" link you don't get anything meaningful, just bland assurances and pretty animations, i.e. entirely non-nerdy. Nowhere do I see basic technical info like "It's a new IP transport protocol used in place of TCP", or "It's compatible with existing networks because...", or "It's (better than|different to) (HTTP/2.0|QUIC|SPDY) because...".
The discussion here, while short, has been much more informative than the site itself. I don't see any good reason to dumb-down the site like that - you're selling a protocol layer, so you're not going to have any non-nerdy customers!
Why are resumable downloads done in that layer? I have no trouble with downloads resuming across network switches when handled by higher-level protocols like chunked HTTP or even FTP.
Similarly, why would you even think about bundling multiple resources in one transfer at that level? Surely that's a higher level thing (like HTTP/2.0), and thus independent of transport? There appears to be a lot of layer-leakage going on - not that it's necessarily bad, but the reasons for it need to be clearer.
Thanks. I'll take this positive beginning at face value :-)
"Nowhere do I see basic technical info like "It's a new IP transport protocol used in place of TCP", or "It's compatible with existing networks because...", or "It's (better than|different to) (HTTP/2.0|QUIC|SPDY) because..."
I'm sorry if the material on the site is not technical enough for your taste. We've just launched and consider the site and the blog a work in progress. (Follow us on twitter at @packetzoom and me personally at @IAmChetanAhuja for updates).
Having said that, our customers are not necessarily people following the network protocol worlds closely. The stuff that you find "bland" and "pretty" is actually very helpful for a large portion of our target audience.
Now of course, those who click on the "Learn" link and don't find enough in-depth info are welcome to, and do, contact us directly. Every single mail we get on info@ or support@ addresses is read by a human in our (rather small) team and yes, we respond. Just as I'm responding to more detailed technical questions here.
"You're selling a protocol layer, so you're not going to have any non-nerdy customers!"
No we're not selling "a protocol layer". We're selling a drop-dead simple way to speed up downloads. Some of our customers are more interested in details of the protocol than others. But the majority of them want their apps to be fast and couldn't care less about, or even want to know about QUIC/SPDY/HTTP2.0 or whatever. Their questions are more like -- How easy is it to integrate? (very). Do I have to rewrite my code? (No). What do I have to change on the server side? (nothing).
And those of our customers who do want to go in-depth into details of the technology, do. They just talk to us and we explain what we're doing to the extent it doesn't reveal trade secrets.
"a lot of layer-leakage going on - not that it's necessarily bad, but the reasons for it need to be clearer"
This seems to be the crux of your issues with what we're doing. The reasons are ultra clear to us. Apps are faster and users are happier. That the only thing that matters.
We've found that merging layers allows us to handle, say, out of order delivery or dropped packets more smartly and more naturally. For far too long, user experience has been held hostage to diktats from the ivory tower. It's about time we start thinking from the ground up to create technology suited to what the users needs and not because we want to maintain some platonic ideal of "layer separation".
Weirdly, if google thinks you're a dodgy sender they won't display the button, which seems counterproductive to me.