Its not just historically (atleast if you include IoT/Industrial/Medical devices) Microsoft's "Section 52" (IoT security research group) called out this issue in several allocators used in IoT in April 2021 [0][1].
It is, it'll build a few fuzzers hitting different areas[0]. The important function in many of those `.c` files is `FuzzerTestOneInput` which is effectively the entrypoint for a single fuzz test.
Taking a look at x509.c[1] which I believe is the most likely to be able to reach the punnycode parser. (I am not at all familiar with the codebase). You can see that the OpenSSL fuzzer is basically doing a holistic fuzz (I assume the i2d* and d2i* functions exercise the parser), that is its just invoking key entrypoints that in theory can exercise all the rest of the functionality with the correct inputs.
Hanno's fuzzer on the other hand, is explicitly only testing the `ossl_punnycode_decode` function[3].
Given the breadth of the fuzzer, I think its very possible OSS-Fuzz just didn't hit it.
So "pentesting" often kinda defaults to "network pentesting" which is closer to an IT job than a software engineering job and its salary range (in general, as you specialize pay goes up regardless).
But there is "application penetration testing" and just application security in general which tends to pay competitively with software engineering. And of course plenty of people do both at the same job.
So pentesting can be competitive but it depends on definitions a bit. That said on the upper end, software dev tends to have more chances to get a big exit by being part of building something. In security you might be with a consulting firm where you have a slight chance at that, but its not common for a security guy to have that sort of big exit.
This is a bit of a common trap, the idea that to do anything you must know everything. When you read writeups you see people just going from some bug to exploit and incorporating obscure bits of knowledge to make it happen. It feels like they must know everything. The reality is they probably spend hours or days banging their head against a wall having an intuition that _something_ is wrong but no idea how to abuse it or that there must be something. Spending hours researching until they can connect the dots. Those hours of frustration are not captured very well in most writeups.
> I know for stuff like this the key is to just get started, and the understanding will follow, but I'm curious if anyone has any recommendations for how to do that.
The single tip I give anyone getting started is:
Follow all the rabbit holes.
Seriously, all of them. Any time you have some random question come up, "Would doing X be vulnerable", "Could I exploit Y feature", "Why didn't this writeup author do Z", "How does A work", "Why send B this way instead of this way" ... all of them. When you have the question, just go spend the time to figure it out. Every rabbit hole you go down, even if it ends up being a dead end, is adding bits and pieces to your knowledge. Over time you build up an immense library of random bits of knowledge that you can draw from in the future.
While I wrote that with an eye towards doing binary-level exploit development against modern targets, the advice for doing manual auditing is pretty universal. It's like how to learn to program you actually have to write code, reading about writing code isn't enough. Practice against anything can be useful.
I'll also leave you my favorite vuln research quote:
"Frustration is a key part of exploit research and you must embrace it accordingly"
Funnily, both ExpressVPN and NordVPN which you call out have been externally audited.
NordVPN had the clients audited by VerSprite last year, and their No-log policy audited by PwC in 2018 and 2020. And a bug bounty program on HackerOne. [1]
ExpressVPN - Windows Client was just audited by F-Secure in March, and server side audits by Cure54, and PwC in 2021 and 2019 respectively. And a bug bounty program on Bug Crowd. [2]
---
For comparison
Mullvad has been audited (Client security and Infrastructure (for privacy)) by Cure53 through 2020, and first was in 2018. Has no bug bounty, but they do still have a vulnerability disclosure program. [3]
ProtonVPN, audits of the no-log policy in April, and clients in 2020. And they run their own bug bounty program.[4]
---
I actually find it kinda interesting that while they've all had audits regarding privacy on the server side, only ExpressVPN has had a security audit of server side components. (Granted I've not look that deeply at this)
[1] Annoying, you can only download the audit reports if you Login then click Reports in the menu
Unfortunately many parts of this paper are beyond me, but this study does appear to find that community level masking has an impact, but it is a study that used actual communities and tracked compliance (and impact of different interventions on encouraging making)
AFAIK, this is the only really good study of mask-wearing. It's large (300k people!), and it's randomised. It tried two kinds of masks, "surgical" and "cloth":
> We used high-quality surgical masks that had had a filtration efficiency of 95% [standard deviation (SD) = 1%]; this is substantially higher than the filtration efficiency of the cloth masks we designed, which had a filtration efficiency of 37% (SD = 6%). These cloth masks had substantially higher filtration than common commercial 3-ply cotton masks, but lower than hybrid masks that use materials not commonly available for community members in low-resource settings (54).
The key result:
> We find clear evidence that surgical masks lead to a relative reduction in symptomatic seroprevalence of 11.1% (aPR = 0.89 [0.78,1.00]; control prevalence = 0.81%; treatment prevalence = 0.72%). Although the point estimates for cloth masks suggests that they reduce risk, the confidence limits include both an effect size similar to surgical masks and no effect at all. (aPR = 0.94 [0.78,1.10]; control: 0.67%; treatment: 0.61%).
It's a shame that even with such a large study, the phenomenon is so noisy that we can't get a strong signal here. But i think the takeaway is that masking has an effect, but it is rather small.
There's a lot more detail in the paper. Including this gem:
> We find no evidence that any of our village-level or household-level treatments, other than mask color, impacted mask-wearing. For mask-color, we see marginally significant differences, small in magnitude. In surgical mask villages, blue masks were more likely to be observed than green (adjusted percentage point difference = 0.03, [-0.00,0.06]), and in cloth mask villages, red more likely than purple (adjusted percentage point difference = -0.02, [-0.04,-0.00]). Text message reminders, incentives for village-leaders, or explicit commitment signals explain little of the observed increase in mask-wearing. Compared to self-protection messaging alone, altruistic messaging had no greater impact on mask-wearing, and twice-weekly text messages and a verbal commitment had no significant effects.
Actually look at the peer review of that study done by Cornell and released this December. When the actual data was analyzed it showed masks had no effect.
On a whole what I found more interesting here was just the techniques they came across through fuzzing that had some impact. Yes its interesting to see the specific combinations that were impacted, but in the real-world there are so many other potential combinations.
The dominate method for request smuggling as of the last few years has been with `Content-Length` and `Transfer-Encoding`. What I found most interesting and the biggest take-away as someone who has worked doing web-app assessments is more just the attacks that they found to work and cause problems.
I mean the details about particular server pairs having issues is great information, as is the fuzzing setup (great use of differential fuzzing) but I think more important is being able to take these potential attack avenues that they had success with and running them against your own deployments. Given how many applications internally are running their own stacks there is still a lot of room for potential issues. I can imagine people running with some of these for bounties in the near future.
A brief summary of the manipulations they had some success with are on pages 7-8. Though if you don't feel like reading, the "headline" version of the list gives you a pretty decent idea of what was having an impact:
So, disclosure policy is kinda an active discussion within the security community but there is a general move away from coordinated disclosure (aka responsible disclosure) where the vendor and reporter coordinated on disclosing the vulnerability, not publicly disclosing until the vendor okays it.
Coordinated disclosure puts a lot of power in the vendor to simply ignore or delay fixing issues, and frankly may not actually be the "responsible" course of action. Full disclosure, where the first warning anyone has about the issue is when all the details are dropped about it to the public _may_ result in a faster patch time but it also increases risk of it being weaponized during that in-between period. There is the chance it was being used in-the-wild without being known also, but releasing the information increases the risk of those in the-wild-attacks but reducing the overhead necessary to carry them out.
All that said, there is a newer option that has been pretty steadily gaining popularity over the last seven or so years. Deadline-based disclosure. This did exist before, but really gained popularity in recent years as Google's Project Zero adopted it as their disclosure policy. This is the idea where the reporter discloses a vulnerability which starts a countdown to the public disclosure (90 days is fairly common, but I've seen 30, 60, and 180 reasonably often also)
I think this deadline-based disclosure option strikes a good balance between the benefits of coordinated, and full disclosures.
Kinda on a related note I found myself needing to make a bunch of these sorts of scraped feeds. The problem for me was the lack of date parsing support which I sorely needed (and it doesn't appear like this option supports it either)
I did write it specifically for my use-case so there are some "warts" on it like custom generators for HackerOne and Google's Monorail bug tracker. But perhaps someone else might benefit from its ability to create slightly more complicated RSS, Atom, or JSON feeds.
So kinda a quick summary given the available information that I've seen.
Back in August 2020 some research was performed looking at introducing vulnerabilities into the Linux Kernel.[0] The paper indicates that three patches were submitted via anonymous gmail accounts to the mailing list and were never committed. The reviewers were provided a proper patch upon accepting the vulnerable one and received explicit confirmation that the maintainers would not move forward with the vulnerable patch.
I'm not sure when exactly questions started being raised about that research. Though I first became aware of it in December. The discussion was mostly around the human involvement and led to the prepublication of the paper being removed and clarifications being issued.[1]
Fast-forward to April 2021. The patch seems to have kicked things off[2]. This was called out as being an impossible situation, and as being a "known-invalid patch" by Greg KH [3].
It appears that at least three patches by this same author introduced vulnerabilities[4] according to Leon Romanovsky. Though I don't have links to the specific patches.
Leading to U.Mn's ban from contributing to the kernel by Greg KH[5]
---------
What is in my opinion unclear at least to me is whether these more recent patches are actually in bad faith or just simply bad. The prevailing theory is that they are part of more research into introducing vulnerabilities. As already stated though, that research and its paper were done in August of 2020. The more recent commits, the official story from Kangjie Lu, Qiushi Wu, and Aditya Pakki[6] are that they are part of "a new project that aims to automatically identify bugs introduced by other patches". This does somewhat align with statements[7] made indicating that the commits were from a static analysis tool being researched which was made prior to this blowing up. Though I will note that the author of that patch was _not_ one of the authors of the apology letter, so may genuinely be unrelated.
This tool story was not believed by Greg KH[8].
And his take is the one that has gained a lot of adoption. That these patches were intentionally made in bad faith for another paper.
I will state that the newer patches that caused problems did _not_ follow the methodology that the original research followed to try to prevent vulnerabilities from actually being introduced into the repo. The original paper, while certainly had issues with methodology and experimenting on people inappropriately, did take steps to prevent any actual vulnerabilities from being committed, whereas the ones in question did not, and even made it to stable branches.
If the official story from U.Mn is true the commits should also have been noted as having been found by a tool, and followed the proper procedure for that, which they did not do. Though it does appear that the vast majority of the patches that were reverted were legitimate patches.[9] Atleast spot checking replies on that mailing list.
I mean on a whole the original research was questionable, but I kind of want to be more charitable in my interpretation of the more recent events but honestly that original patch that kicked things off is pretty bad.
I have followed the situation somewhat closely as well and there seem to be a lot of misinformation being thrown around. As far as I can tell (correct me if I'm wrong), no actual malicious commit has been found to be deliberately introduced and merged into the tree, and most of the recent commits that triggered this whole mess were from a claimed custom static analyzers that resulted in wrong/useless patches but not necessarily malicious.
I honestly think the situation is somewhat overblown, and some maintainers think so as well. To quote Jason Gunthorpe:
> So, this revert is based on not trusting the authors to carry out their work in the manner they explained? From what I've reviewed, and general sentiment of other people's reviews I've read, I am concerned this giant revert will degrade kernel quality more than the experimenters did - especially if they followed their stated methodology.
and Doug Ledford:
> I have to agree with Jason. This seems like trying to push a thumbtack
into a bulletin board using a pyle driver. Unless the researchers are
lying (which I've not seen a clear indication of), the 190 patches you
have selected here are nothing more than collateral damage while you are
completely missing the supposed patch submission addresses from which
the malicious patches were sent!
> As far as I can tell (correct me if I'm wrong), no actual malicious commit has been found to be deliberately introduced and merged into the tree, and most of the recent commits that triggered this whole mess were from a claimed custom static analyzers that resulted in wrong/useless patches but not necessarily malicious.
My understanding is that the malicious commits described in their paper were submitted under alias email addresses, and the authors have not identified those addresses or the commits. So at this point there is no way to confirm that these malicious commits were properly reverted besides taking the authors word for it. To quote Mike Dolan's letter: "While the U of MN researchers claimed to take steps to prevent inclusion of vulnerabilities in the final software, their failure to gain consent suggests a lack of care. There are also amplified consequences because Linux kernel changes are picked up by many other downstream projects that build off of the kernel codebase." And I think it's fair for maintainers to question the competence and want to be able to verify everything, especially considering the consequences if the authors made a mistake.
But yes, assuming they used fake accounts, none of those 190 or so patches selected are the malicious ones from the paper. None of them appear to introduce any vulnerabilities, and the same for the weird commits from Aditya.
[0] https://msrc-blog.microsoft.com/2021/04/29/badalloc-memory-a...
[1] Full list: https://www.cisa.gov/uscert/ics/advisories/icsa-21-119-04